Shooting control apparatus, shooting control method, and shooting apparatus

ABSTRACT

There is provided a shooting control apparatus, a shooting control method for controlling operations of a shooting part mounted on a mobile object such as automobile, and a shooting apparatus mounted on a mobile object for use. A center region includes fine pixels, for high-resolution shooting. On the other hand, peripheral regions includes large-size pixels or are at a high sensitivity by pixel addition, thereby reducing blur or focal plane distortion by shortening the exposure time when fast driving or shooting a moving object. Further, while traveling during the nighttime or in a dark place, the peripheral regions not irradiated by the headlamps can be shot in a long exposure time and at a sufficient sensitivity.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Phase of International PatentApplication No. PCT/JP2017/009554 filed on Mar. 9, 2017, which claimspriority benefit of Japanese Patent Application No. JP 2016-088783 filedin the Japan Patent Office on Apr. 27, 2016. Each of theabove-referenced applications is hereby incorporated herein by referencein its entirety.

TECHNICAL FIELD

The technology disclosed in the present specification relates to ashooting control apparatus and a shooting control method for controllingoperations of a shooting part, as well as a shooting apparatus, andparticularly to a shooting control apparatus and a shooting controlmethod for controlling operations of a shooting part mounted on a mobileobject such as automobile, as well as a shooting apparatus mounted on amobile object for use.

BACKGROUND ART

In recent years, camera-equipped automobiles have increased (see PatentDocument 1, for example). Images shot by the vehicle-mounted camera canbe recorded on a dashboard camera or images shot by the vehicle-mountedcamera can be used for traveling support or eyesight support. Forexample, an image shot by the vehicle-mounted camera is processed tosense the headlamps of an oncoming vehicle or the tail lamps of aleading vehicle and to detect information on other vehicles therearoundwhile traveling during the nighttime.

CITATION LIST Patent Document

-   Patent Document 1: Japanese Patent Application Laid-Open No.    2013-115625-   Patent Document 2: Japanese Patent Application Laid-Open No.    2006-148496-   Patent Document 3: Japanese Patent Application Laid-Open No.    2011-130022-   Patent Document 4: Japanese Patent Application Laid-Open No.    2014-204149-   Patent Document 5: Japanese Patent Application Laid-Open No.    2014-155175-   Patent Document 6: Japanese Patent Application Laid-Open No.    2014-165520-   Patent Document 7: Japanese Patent Application Laid-Open No.    2008-131580

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

The technology disclosed in the present specification is directed toproviding a shooting control apparatus and a shooting control method forcontrolling operations of a shooting part mounted on a mobile objectsuch as automobile, as well as a shooting apparatus mounted on a mobileobject for use.

Solutions to Problems

The technology disclosed in the present specification is made inconsideration of the above object, and a first aspect thereof is ashooting control apparatus including: a control part configured tocontrol shooting conditions of a center region in a shooting part havinga plurality of pixels to be any of a higher sensitivity, a higher framerate, a shorter exposure time, and a higher operation frequency thanperipheral regions in the shooting part.

According to a second aspect of the technology disclosed in the presentspecification, the control part of the shooting control apparatusaccording to the first aspect sets the peripheral regions at a highersensitivity than the center region.

According to a third aspect of the technology disclosed in the presentspecification, the control part of the shooting control apparatusaccording to the first aspect sets the peripheral regions at a shorterexposure time than the center region.

According to a fourth aspect of the technology disclosed in the presentspecification, the control part of the shooting control apparatusaccording to the first aspect sets the peripheral regions at a longerexposure time than the center region.

According to a fifth aspect of the technology disclosed in the presentspecification, the control part of the shooting control apparatusaccording to the first aspect sets the peripheral regions at a higherframe rate than the center region.

According to a sixth aspect of the technology disclosed in the presentspecification, the control part of the shooting control apparatusaccording to the first aspect performs a signal processing in theperipheral regions at a higher operation frequency than in the centerregion.

According to a seventh aspect of the technology disclosed in the presentspecification, the control part of the shooting control apparatusaccording to the first aspect sets the peripheral regions at a highersensitivity and a higher frame rate than the center region.

According to an eighth aspect of the technology disclosed in the presentspecification, the control part of the shooting control apparatusaccording to the first aspect sets the peripheral regions at a highersensitivity than the center region and at the same frame rate as thecenter region.

According to a ninth aspect of the technology disclosed in the presentspecification, the control part of the shooting control apparatusaccording to the first aspect sets the peripheral regions at a highersensitivity than the center region and at the same or a longer exposuretime as or than the center region.

According to a tenth aspect of the technology disclosed in the presentspecification, the control part of the shooting control apparatusaccording to the second aspect performs pixel addition reading orthinning reading on the peripheral regions.

According to an eleventh aspect of the technology disclosed in thepresent specification, the control part of the shooting controlapparatus according to the first aspect controls shooting conditions ofthe peripheral regions relative to the center region depending on aplace where the shooting part is mounted on a vehicle or a drivingsituation of the vehicle.

According to a twelfth aspect of the technology disclosed in the presentspecification, the control part of the shooting control apparatusaccording to the first aspect controls at least one of the position, theshape, and the size of the center region depending on a place where theshooting part is mounted on a vehicle or a driving situation of thevehicle.

According to a thirteenth aspect of the technology disclosed in thepresent specification, the control part of the shooting controlapparatus according to the first aspect controls at least one of thenumber of phases of the peripheral regions, the position, the shape, andthe size of each peripheral region depending on a driving situation of avehicle mounting the shooting part thereon.

In addition, a fourteenth aspect of the technology disclosed in thepresent specification is a shooting control method including: a controlstep of controlling shooting conditions of a center region in a shootingpart having a plurality of pixels to be any of a higher sensitivity, ahigher frame rate, a shorter exposure time, and a higher operationfrequency than peripheral regions in the shooting part.

In addition, a fifteenth aspect of the technology disclosed in thepresent specification is a shooting apparatus including: an imagingdevice including a center region, and peripheral regions configured oflarger-size pixels than the center region.

According to a sixteenth aspect of the technology disclosed in thepresent specification, in the shooting apparatus according to thefifteenth aspect, each of the center region and the peripheral regionsis scanned in parallel.

According to a seventeenth aspect of the technology disclosed in thepresent specification, the shooting apparatus according to the fifteenthaspect further includes: a signal processing part configured to performpixel reading and an AD conversion processing for each of the centerregion and the peripheral regions.

According to an eighteenth aspect of the technology disclosed in thepresent specification, in the shooting apparatus according to thefifteenth aspect, the peripheral regions are set at a shorter exposuretime than the center region.

According to a nineteenth aspect of the technology disclosed in thepresent specification, in the shooting apparatus according to thefifteenth aspect, the peripheral regions are set at a higher frame ratethan the center region.

According to a twentieth aspect of the technology disclosed in thepresent specification, in the shooting apparatus according to thefifteenth aspect, at least one of an exposure time or a frame rate ofthe peripheral regions is controlled relative to the center regiondepending on a driving situation of a vehicle mounting the shootingapparatus thereon.

Effects of the Invention

According to the technology disclosed in the present specification, itis possible to provide a shooting control apparatus and a shootingcontrol method for controlling operations of a shooting part mounted ona mobile object such as automobile, as well as a shooting apparatusmounted on a mobile object for use.

Additionally, the effects described in the present specification aremerely exemplary, and the effects of the present invention are notlimited thereto. Further, the present invention may produce additionaleffects other than the above effects.

Still other objects, characteristics, or advantages of the technologydisclosed in the present specification will be apparent by more detaileddescription based on embodiments described below or the accompanyingdrawings.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram schematically illustrating an exemplaryconfiguration of a vehicle control system 2000 to which the technologydisclosed in the present specification can be applied.

FIG. 2 is a diagram illustrating exemplary installation positions of ashooting part 2410 and a vehicle exterior information detection part2420.

FIG. 3 is a diagram illustrating how respective objects 301 and 302 inan image shot by a vehicle-mounted camera 300 change as a vehicletravels.

FIG. 4 is a diagram illustrating a landscape in front of a vehicle byway of example.

FIG. 5 is a diagram illustrating an image shot by a vehicle-mountedcamera on a vehicle traveling toward the landscape illustrated in FIG.4.

FIG. 6 is a diagram for explaining characteristics of an image shot by avehicle-mounted camera.

FIG. 7 is a diagram illustrating an exemplary image shot by avehicle-mounted camera on a vehicle traveling during the nighttime or ina dark place.

FIG. 8 is a diagram illustrating a configuration of a pixel region 800of an imaging device applicable to a vehicle-mounted camera.

FIG. 9 is a diagram illustrating how the pixel region 800 is scanned perdivided region.

FIG. 10 is a diagram schematically illustrating an exemplaryconfiguration of a camera module 1000 for reading the respective regions801 to 803 in the pixel region 800 of the imaging device in parallel.

FIG. 11 is a diagram illustrating a shooting range 1100 when theshooting apparatus with the configuration illustrated in FIG. 8 to FIG.10 is installed on the front nose of a vehicle.

FIG. 12 is a diagram illustrating an exemplary timing chart of anexposure/reading processing per region in the pixel region 800.

FIG. 13 is a diagram illustrating a timing chart of the exposureprocessing for one frame.

FIG. 14 is a diagram illustrating other exemplary timing chart of theexposure/reading processing per region in the pixel region 800.

FIG. 15 is a diagram illustrating other exemplary timing chart of theexposure/reading processing per region in the pixel region 800.

FIG. 16 is a diagram illustrating an exemplary installation place of theshooting apparatus on a vehicle.

FIG. 17 is a diagram illustrating exemplary region division of a pixelregion of an imaging device in a shooting apparatus 1601.

FIG. 18 is a diagram illustrating an exemplary installation place of ashooting apparatus on a vehicle.

FIG. 19 is a diagram illustrating exemplary region division of a pixelregion of an imaging device in a shooting apparatus 1801.

FIG. 20 is a diagram illustrating a pixel region 200 divided into threephases of peripheral regions.

FIG. 21 is a diagram illustrating a pixel region 210 divided intorectangular regions by way of example.

FIG. 22 is a diagram illustrating an exemplary imaging device where apixel region is divided by pixel addition.

FIGS. 23A, 23B, and 23C are diagrams for explaining adaptive control ofregion division of a pixel region.

FIGS. 24A, 24B, and 24C are diagrams for explaining adaptive control ofregion division of a pixel region.

FIGS. 25A and 25B are diagrams for explaining adaptive control of regiondivision of a pixel region.

FIGS. 26A and 26B are diagrams for explaining adaptive control of regiondivision of a pixel region.

FIGS. 27A and 27B are diagrams for explaining adaptive control of regiondivision of a pixel region.

FIGS. 28A and 28B are diagrams for explaining adaptive control of regiondivision of a pixel region.

FIG. 29 is a flowchart illustrating a processing procedure of performingshooting condition control of a peripheral region depending on a drivingsituation.

FIG. 30 is a flowchart illustrating a processing procedure of performingregion division of a pixel region and shooting condition control of aperipheral region depending on a driving situation.

FIG. 31 is a diagram illustrating an exemplary configuration of a cameramodule 3100 applicable as a vehicle-mounted camera.

FIG. 32 is a diagram illustrating an exemplary configuration of animaging device 3120.

FIG. 33 is a diagram illustrating an exemplary configuration of a pixelregion configured of uniform-size pixels.

FIG. 34 is a diagram illustrating an exemplary configuration of a pixelregion configured of uniform-size pixels.

FIG. 35 is a diagram illustrating an exemplary configuration of a pixelregion configured of uniform-size pixels.

MODE FOR CARRYING OUT THE INVENTION

Embodiments of the technology disclosed in the present specificationwill be described below in detail with reference to the drawings.

A. System Configuration

FIG. 1 schematically illustrates an exemplary configuration of a vehiclecontrol system 2000 to which the technology disclosed in the presentspecification can be applied. The illustrated vehicle control system2000 is configured of a plurality of control units such as a drivesystem control unit 2100, a body system control unit 2200, a batterycontrol unit 2300, a vehicle exterior information detection unit 2400, avehicle interior information detection unit 2500, and an integratedcontrol unit 2600.

The respective control units 2100 to 2600 are mutually connected via acommunication network 2010. The communication network 2010 may be avehicle-mounted communication network conforming to any communicationstandard such as controller area network (CAN), local interconnectnetwork (LIN), local area network (LAN), or FlexRay (registeredtrademark), or a network conforming to a locally-defined communicationstandard, for example.

Each of the control units 2100 to 2600 includes a microcomputerconfigured to perform computation processings according to variousprograms, a storage part configured to store programs executed by themicrocomputer, parameters used for various computations, or the like,and a drive circuit configured to drive various apparatuses to becontrolled, for example. Further, each of the control units 2100 to 2600includes a network interface (IF) configured to make communication withother control units via the communication network 2010, and includes acommunication interface configured to make wired communication orwireless communication with apparatuses, sensors, or the like outsidethe vehicle.

The drive system control unit 2100 controls operations of apparatusesfor vehicle drive system according to various programs. For example, thedrive system control unit 2100 functions as a control apparatus of adriving force generation apparatus configured to generate a vehicledriving force such as internal combustion engine or drive motor, adriving force transmission mechanism configured to transmit a drivingforce to the wheels, a steering mechanism configured to adjust asteering angle of the vehicle, a braking apparatus configured togenerate a braking force of the vehicle, and the like. Further, thedrive system control unit 2100 may include functions as a controlapparatus such as antilock brake system (ABS) or electronic stabilitycontrol (ESC).

A vehicle state detection part 2110 is connected to the drive systemcontrol unit 2100. The vehicle state detection part 2110 includes atleast one of a gyro sensor configured to detect an angular speed ofaxial rotation of the vehicle body, an acceleration sensor configured todetect an acceleration of the vehicle, and a sensor configured to detectthe operation amount of the acceleration pedal, the operation amount ofthe brake pedal, a steering angle of the steering wheel, enginerevolutions, a rotation speed of the wheels, or the like, for example.The drive system control unit 2100 performs a computation processing byuse of a signal input from the vehicle state detection part 2110, andcontrols the internal combustion engine, the drive motor, the electricpower steering apparatus, the brake apparatus, and the like (none ofwhich is illustrated).

The body system control unit 2200 controls operations of variousapparatuses mounted on the vehicle body according to various programs.For example, the body system control unit 2200 functions as a controlapparatus configured to lock and unlock the doors and to start and stopthe system 2000 such as keyless entry system or smart key system, or acontrol apparatus for a power window apparatus or various lamps(including headlamps, tail lamps, brake lamp, turn signals, and foglamp) (is assumed to include a function of switching the headlampsbetween high beam and low beam). When a radio wave sent from a portabletransmitter incorporated in a key (or instead of a key) or a signal fromvarious switches arrives at the body system control unit 2200, the bodysystem control unit 2200 controls the door lock apparatus, the powerwindow apparatus, the lamps, and the like of the vehicle (none of whichis illustrated in FIG. 1).

The battery control unit 2300 controls a secondary battery as a powersupply source of the drive motor according to various programs. Forexample, in the battery control unit 2300, a battery apparatus 2310including a secondary battery measures a battery temperature, a batteryoutput voltage, a battery remaining capacity, and the like of thesecondary battery, and outputs them to the battery control unit 2300.The battery control unit 2300 performs the computation processing by useof the information input from the battery apparatus 2310, and controlstemperature adjustment of the secondary battery, or controls a coolingapparatus (not illustrated) and the like provided in the batteryapparatus 2310.

The vehicle exterior information detection unit 2400 detects informationon the exterior of the vehicle mounting the vehicle control system 2000thereon. For example, at least one of a shooting part 2410 or a vehicleexterior information detection part 2420 is connected to the vehicleexterior information detection unit 2400.

The shooting part 2410 is what is called a vehicle-mounted camera, andincludes at least one of a time of flight (ToF) camera, a stereo camera,a monocular camera, an infrared camera, and other camera. According tothe technology disclosed in the present specification, the shootingoperations of the shooting part 2410 are dynamically controlleddepending on a driving situation or the like. The shooting operations tobe controlled include sensitivity, exposure time, frame rate, and thelike. Controlling the shooting operations will be described below indetail. Additionally, an exposure time described below indicates a timein which the shutter is opened and the imaging device is exposed tolight (or exposed) during shooting, and is synonymous with shutter speed(SS) (a short exposure time corresponds to a high shutter speed, and along exposure time corresponds to a low shutter speed). Further, a framerate is the number of frames processed per unit time, and is generallyexpressed in a unit of frame per second (fps) indicating a numericalvalue per second. A frame rate of the display apparatus is the number offrames switched per unit time, and a frame rate during moving pictureshooting by the shooting apparatus is the number of frames shot per unittime. A “high” frame rate indicates a short interval between frames.Thus, a high frame rate is synonymous with “fast continuous shooting”.

The vehicle exterior information detection part 2420 includes at leastone of an environment sensor configured to detect current weather ormeteorological phenomenon, a surrounding information detection sensorconfigured to detect a peripheral vehicle, an obstacle, a pedestrian,and the like, and a speech sensor (microphone configured to collectsounds generated around the vehicle) (none of which is illustrated), forexample. In a case where the vehicle exterior information detection part2420 is a speech sensor, sounds outside the vehicle along with accidentor near-miss, such as horn, sudden braking, and collision sound, can beacquired.

An environment sensor described herein is a raindrop sensor configuredto detect rainy weather, a fog sensor configured to detect fog, asunshine sensor configured to detect a degree of sunshine, a snow sensorconfigured to detect snowfall, or the like, for example. Further, asurrounding information detection sensor is configured of an ultrasonicsensor, a radar apparatus, alight detection and ranging, laser imagingdetection and ranging (LIDAR) apparatus, or the like.

The shooting part 2410 and the vehicle exterior information detectionpart 2420 may be configured as an independent sensor or apparatus,respectively, or may be configured as an apparatus in which a pluralityof sensors or apparatuses are integrated. The installation positions ofthe shooting part 2410 and the vehicle exterior information detectionpart 2420 will be described below in detail.

The vehicle exterior information detection unit 2400 causes the shootingpart 2410 to shoot an image of the exterior of the vehicle, and receivesthe shot image data from the shooting part 2410. Further, the vehicleexterior information detection unit 2400 receives detected informationfrom the vehicle exterior information detection part 2420. In a casewhere the vehicle exterior information detection part 2420 is anultrasonic sensor, a radar apparatus, or a LIDAR apparatus, the vehicleexterior information detection unit 2400 originates an ultrasonic wave,an electromagnetic wave, or the like, and receives information on areflected wave from the vehicle exterior information detection part2420.

The vehicle exterior information detection unit 2400 may perform animage recognition processing of recognizing, for example, surroundingperson, vehicle, obstacle, road sign (road guidance) installed along aroad, or road sign drawn on a road, an object recognition processing ofdetecting or recognizing an object outside the vehicle, and a processingof detecting a distance to an object outside the vehicle on the basis ofthe information received from the vehicle exterior information detectionpart 2420. Further, the vehicle exterior information detection unit 2400may perform an environment recognition processing of recognizing asurrounding environment such as rainfall, fog, or state of road on thebasis of the information received from the vehicle exterior informationdetection part 2420.

Additionally, the vehicle exterior information detection unit 2400 mayperform a distortion correction, positioning processing, or the like onthe image data received from the vehicle exterior information detectionpart 2420. Further, the vehicle exterior information detection unit 2400may generate a perspective image or panorama image by combining theimage data shot by different shooting parts 2410. Further, the vehicleexterior information detection unit 2400 may perform a viewpointconversion processing by use of the image data shot by differentshooting parts 2410.

The vehicle interior information detection unit 2500 detects informationon the interior of the vehicle. The vehicle interior informationdetection unit 2500 is connected with a vehicle interior state detectionpart 2510 configured to detect a state of the driver driving thevehicle, for example, and detects information on the interior of thevehicle on the basis of the driver's state information input from thevehicle interior state detection part 2510. A driver described herein isa passenger seated on the driver seat in the vehicle among thepassengers inside the vehicle, or a passenger who is stored as a personto drive by the integrated control unit 2600.

For example, the vehicle interior information detection unit 2500 maycalculate a degree of fatigue or a degree of concentration of thedriver, or determines whether the driver is falling asleep. Further, thevehicle interior information detection unit 2500 detects variousdriver's states, and determines whether the driver (or a passenger otherthan the driver) can drive the vehicle. The vehicle interior informationdetection unit 2500 may sense the driver on the basis of the positionswhere the passengers are seated, or may determine the driver bycomparing a face image previously registered as a driver with shot faceimages on the basis of the faces of the passengers included in the imageshooting the interior of the vehicle.

The vehicle interior state detection part 2510 may include avehicle-mounted camera (Dramoni camera) configured to shoot the interiorof the vehicle such as the driver or other passenger, a biologicalsensor configured to detect biological information of the driver, amicrophone configured to collect sounds inside the vehicle, or the like.Facial authentication of the driver or other passenger can be performedby facial recognition of an image shot by the Dramoni camera. Further, apoint of gaze (or an eye direction) of the driver can be detected on thebasis of a direction in which the recognized face directs or a motion ofthe eyes included in the recognized face. The biological sensor isprovided on the seat, the steering wheel, or the like, for example, anddetects biological information on the driver seated on the driver seator the driver gripping the steering wheel. Further, the microphone canacquire sounds inside the vehicle along with accident or near-miss suchas horn, sudden braking, or speech (scream) of a passenger. The vehicleinterior information detection unit 2500 may perform a signal processingsuch as noise canceling on a speech signal collected by the microphone.The vehicle interior information detection unit 2500 may modulate speechother than specific speech (such as driver's or previously-registeredvoice) in order to protect privacy, for example.

Further, the vehicle interior state detection part 2510 may include aload sensor configured to detect a load (whether or not a person isseated on the seat) applied on the driver seat or the other seats (suchas the front passenger seat and the rear passenger seats). Further, thevehicle interior state detection part 2510 may detect a driver's stateon the basis of operations on various devices by which the driveroperates the vehicle such as accelerator, brake, steering wheel,windshield wipers, turn signals, air conditioner, and other switches.Further, the vehicle interior state detection part 2510 may check astatus such as whether the driver has his/her driver's license orwhether the driver refuses to drive.

The integrated control unit 2600 controls the total operations in thevehicle control system 2000 according to various programs. In theexample illustrated in FIG. 1, the integrated control unit 2600 includesa microcomputer 2610, a general-purpose communication interface 2620, adedicated communication interface 2630, a positioning part 2640, abeacon reception part 2650, an in-vehicle device interface 2660, aspeech/image output part 2670, a vehicle-mounted network interface 2680,and a storage part 2690. Further, the integrated control unit 2600 isconnected with an input part 2800.

The input part 2800 is configured of an apparatus which the driver orother passenger can operate for input, such as touch panel, button,microphone, switch, or lever, for example. The input part 2800 may be aremote control apparatus using infrared ray or other radio wave, or maybe an externally-connected device such as cell phone, personal digitalassistant (PDA), Smartphone, or tablet terminal corresponding to theoperations of the vehicle control system 2000 (none of which isillustrated), for example. The input part 2800 may be operated by speechinput via a microphone. The input part 2800 may be a camera, forexample, and in this case, a passenger can input information into theintegrated control unit 2600 by his/her gesture. Further, the input part2800 may include an input control circuit or the like configured togenerate an input signal on the basis of the information input by thepassenger or the like by use of the input part 2800, for example, and tooutput it to the integrated control unit 2600. The passengers includingthe driver can input various items of data into the vehicle controlsystem 2000 or can give an instruction on a processing operation byoperating the input part 2800.

The storage part 2690 may include a random access memory (RAM)configured to store various programs executed by the microcomputer, oran electrically erasable and programmable read only memory (EEPROM)configured to store various parameters, calculation results, sensors'detected values, and the like. Further, the storage part 2690 mayinclude a large-capacity storage apparatus (not illustrated) configuredof a magnetic storage device such as hard disc drive (HDD), asemiconductor storage device such as solid state drive (SSD), an opticalstorage device, a magnetooptical storage device, or the like. Thelarge-capacity storage apparatus can be used to record (as a dashboardcamera) videos around the vehicle or inside the vehicle shot by theshooting part 2410, for example.

The general-purpose communication interface 2620 is a general-purposecommunication interface configured to mediate communication with variousdevices present in the external environment. The general-purposecommunication interface 2620 mounts a cellular communication protocolsuch as global system of mobile communications (GSM) (registeredtrademark), WiMAX, long term evolution (LTE), or LTE-advanced (LTE-A),wireless LAN such as Wi-Fi (registered trademark), or other wirelesscommunication protocol such as Bluetooth (registered trademark). Thegeneral-purpose communication interface 2620 can connect to a device(such as application server, control server, management server, or thelike) present on an external network (such as Internet, Cloud network,or provider-specific network) via a base station in the cellularcommunication, an access point in the wireless LAN, or the like, forexample. Further, the general-purpose communication interface 2620 mayconnect with a terminal present near the vehicle (such as an informationterminal owned by the driver or a pedestrian, a shop terminal installedin a shop adjacent to a road on which the vehicle is traveling, amachine type communication (MTC) terminal connected to a communicationnetwork not via a person (such as gas meter for home use or automaticvendor), or the like) by use of the peer to peer (P2P) technology, forexample.

The dedicated communication interface 2630 is a communication interfaceconfigured to support a communication protocol defined for use in thevehicle. The dedicated communication interface 2630 may mount a standardprotocol such as wireless access in vehicle environment (WAVE) as acombination of the lower-layer IEEE 802.11p and the higher-layer IEEE1609, dedicated short range communications (DSRC), or cellularcommunication protocol, for example. The dedicated communicationinterface 2630 typically makes V2X communication as a concept includingone or more of vehicle to vehicle communication, vehicle toinfrastructure communication, vehicle to home communication, and vehicleto pedestrian communication.

The positioning part 2640 receives a global navigation satellite system(GNSS) signal from the GNSS satellite (such as a global positioningsystem (GPS) signal from the GPS satellite), for example, to performpositioning, and generates position information including the latitude,longitude, and altitude of the vehicle. Additionally, the positioningpart 2640 may specify a current position on the basis ofelectronically-measured information from a wireless access point by useof PlaceEngine (registered trademark), or may acquire positioninformation from a portable terminal of a passenger such as cell phone,personal handy-phone system (PHS), or Smartphone having a positioningfunction.

The beacon reception part 2650 receives a radio wave or electromagneticwave originated from a wireless station installed on a road, and thelike, for example, and acquires a current position of the vehicle, orroad traffic information (information on traffic jam, road blocked,required time, or the like). Additionally, the functions of the beaconreception part 2650 can be included in the dedicated communicationinterface 2630 to be mounted.

The in-vehicle device interface 2660 is a communication interfaceconfigured to mediate connection between the microcomputer 2610 andvarious devices 2760 present inside the vehicle. The in-vehicle deviceinterface 2660 may establish wireless connection by use of a wirelesscommunication protocol such as wireless LAN, Bluetooth (registeredtrademark), near field communication (NFC), or wireless universal serialbus (USB) (WUSB). Further, the in-vehicle device interface 2660 mayestablish wired connection of USB, high definition multimedia interface(HDMI) (registered trademark), mobile high-definition link (MHL), or thelike via a connection terminal (and a cable as needed) (notillustrated). The in-vehicle device interface 2660 exchanges controlssignals or data signals with a mobile device or wearable device of apassenger, or an in-vehicle device 2760 installed or attached in thevehicle, for example.

The vehicle-mounted network interface 2680 is an interface configured tomediate communication between the microcomputer 2610 and thecommunication network 2010. The vehicle-mounted network interface 2680exchanges signals and the like according to a predetermined protocolsupported by the communication network 2010.

The microcomputer 2610 in the integrated control unit 2600 controls thevehicle control system 2000 according to various programs on the basisof the information acquired via at least one of the general-purposecommunication interface 2620, the dedicated communication interface2630, the positioning part 2640, the beacon reception part 2650, thein-vehicle device interface 2660, and the vehicle-mounted networkinterface 2680.

For example, the microcomputer 2610 may compute a control target valueof the driving force generation apparatus, the steering mechanism, orthe braking apparatus on the basis of the acquired vehicle interior andexterior information, and output a control command to the drive systemcontrol unit 2100. For example, the microcomputer 2610 may performcooperative control for collision avoidance or collision alleviation ofthe vehicle, follow traveling based on inter-vehicle distance, drivingat kept vehicle speed, automatic driving, and the like.

Further, the microcomputer 2610 may create local map informationincluding peripheral information of a current position of the vehicle onthe basis of the information acquired via at least one of thegeneral-purpose communication interface 2620, the dedicatedcommunication interface 2630, the positioning part 2640, the beaconreception part 2650, the in-vehicle device interface 2660, and thevehicle-mounted network interface 2680. Further, the microcomputer 2610may predict dangers such as collision of the vehicle, approach to apedestrian or building, and entry into a road blocked, or the like onthe basis of the acquired information, and generate an alarm signal. Analarm signal described herein is a signal for issuing an alarm sound orturning on an alarm lamp, for example.

Further, the microcomputer 2610 may realize a dashboard camera functionby use of the storage part 2690 or the like. Specifically, themicrocomputer 2610 may control recording videos around the vehicle orinside the vehicle shot by the shooting part 2410.

The speech/image output part 2670 transmits an output signal of at leastone of speech or image to an output apparatus capable of visually oraurally notifying the passengers in the vehicle or the outside of thevehicle of information. In a case where the output apparatus is adisplay apparatus, the display apparatus visually displays the resultsacquired in various processings performed by the microcomputer 2610 orthe information received from other control unit in various forms suchas text, image, table, and graph. Further, in a case where the outputapparatus is a speech output apparatus, the speech output apparatusconverts an audio signal configured of reproduced speech data, acousticdata, or the like into an analog signal, and aurally outputs the analogsignal. In the example illustrated in FIG. 1, an audio speaker 2710, adisplay part 2720, and an instrument panel 2730 are equipped as outputapparatuses.

The display part 2720 may include at least one of an onboard display anda head-up display, for example. The head-up display is a deviceconfigured to show an image (formed at a point at infinity) within thedriver's eyesight by use of the windshield. The display part 2720 mayinclude an augmented reality (AR) display function. The vehicle may beprovided with headphones, projector, lamp, or the like in addition tothe above items.

Further, the instrument panel 2730 is arranged in front of the driverseat (and the front passenger seat), and includes a speedometer ortachometer, a meter panel indicating information required for travelingof the vehicle such as fuel meter, water temperature meter, and distancemeter, or a navigation system for traveling guidance to a destination.

Additionally, at least two control units among a plurality of controlunits configuring the vehicle control system 2000 illustrated in FIG. 1may be integrally configured into one physical unit. Further, thevehicle control system 2000 may further include control units other thanthose illustrated in FIG. 1. Alternatively, at least one of the controlunits 2100 to 2600 may be configured as a physical collection of two ormore units. Further, part of the functions to be served by the controlunits 2100 to 2600 may be realized by other control unit. In otherwords, if the computation processing realized by exchanging informationvia the communication network 2010 is configured to be performed in anycontrol unit, the configuration of the vehicle control system 2000 canbe permitted to change. Further, the sensors or apparatuses connected toany control unit may be connected to other control unit, and informationdetected or acquired by a sensor or apparatus may be mutually exchangedbetween a plurality of control units via the communication network 2010.

FIG. 2 illustrates exemplary installation positions of the shooting part2410 and the vehicle exterior information detection part 2420. In theFigure, shooting parts 2910, 2912, 2914, 2916, and 2918 correspond tothe shooting part 2410, and are arranged on at least one position of thefront node, side mirrors, the rear bumper, the back door of a vehicle2900, and the top of the windshield inside the vehicle, for example. Theshooting part 2910 provided almost at the center of the front nose andthe shooting part 2918 provided at the top of the windshield inside thevehicle mainly capture images in front of the vehicle 2900. A leadingvehicle, a pedestrian, an obstacle, a traffic light, a road sign, alane, and the like can be detected on the basis of the images capturedin front of the vehicle 2900. Further, the shooting parts 2912 and 2914provided on the side mirrors mainly capture images on the sides of thevehicle 2900. Further, the shooting part 2916 provided on the rearbumper or the back door mainly captures images behind the vehicle 2900.

In FIG. 2, a shooting range a indicates a shooting range of the shootingpart 2910 provided almost at the center of the front nose, shootingranges b and c indicate shooting ranges of the shooting parts 2914 and2912 provided on the right and left side mirrors, respectively, and ashooting range d indicates a shooting range of the shooting part 2916provided on the rear bumper or the back door. For example, the imagedata shot by the shooting parts 2910, 2912, 2914, and 2916 areoverlapped thereby to acquire a perspective image of the vehicle 2900viewed from above. Additionally, a shooting range of the shooting part2918 provided at the top of the windshield inside the vehicle isomitted.

The vehicle exterior information detection parts 2920, 2922, 2924, 2926,2928, and 2930 provided at the front, the rear, the sides, and thecorners of the vehicle 2900, and at the top of the windshield inside thevehicle are configured of an ultrasonic sensor or a radar apparatus, forexample. The vehicle exterior information detection parts 2920, 2926,and 2930 provided at the front nose, the rear bumper or the back door ofthe vehicle 2900, and at the top of the windshield inside the vehiclemay be LIDAR apparatuses, for example. The vehicle exterior informationdetection parts 2920 to 2930 are mainly used to detect a leadingvehicle, a pedestrian, an obstacle, or the like.

B. Shooting Control of Vehicle-Mounted Camera

B-1. Images Shot by Vehicle-Mounted Camera

FIG. 2 illustrates the installation positions of the vehicle-mountedcameras by way of example. One of the characteristics of images shot bya camera mounted on a mobile object such as vehicle-mounted camera isthat motions and changes of an object are non-uniform per position inthe screen.

FIG. 3 illustrates how respective objects 301 and 302 in an image shotby the vehicle-mounted camera 300 change as the vehicle travels.

The object 301 which can be captured in an eye direction 311 almostequal to a front direction 310 of the vehicle is rarely different in theeye direction 311 of an object 301′ even after the vehicle travels, andrarely changes in its position on a shot image 320. Further, a change inthe image per frame is small. Thus, the object 301 can be shot to berelatively clear (or at a high resolution) by the vehicle-mounted camera300.

On the other hand, the object 302 captured in an eye direction 312 whichforms a large angle with the front direction 310 of the vehicle isdifferent in the eye direction 312 of an object 302′ after the vehicletravels, and the position on the shot image 302 largely moves from theobject 302 before moving, and the image is easily defocused. When thespeed of the vehicle increases, the amount of movement in the image islarger, and blur or focal plane distortion easily occurs to the object302′ and the object is difficult to recognize. As the vehicle speed ishigher, blur or focal plane distortion is more serious.

For example, assuming that the landscape in front of the vehicle asillustrated in FIG. 4 is shot while the vehicle is traveling (is goingstraight ahead), around the center of the shot image is relativelyclear, but blur of a moving object occurs in the periphery of the shotimage as illustrated in FIG. 5.

An image shot by the vehicle-mounted camera can be divided into a centerregion 601 with a small amount of movement of an object and with a highresolution and a peripheral region 602 with a large amount of movementof an object as illustrated in FIG. 6. An object can be shot relativelyclearly in the center region 601, which does not cause a problem. To thecontrary, the present applicants think that the peripheral region 602needs to be shot in a short exposure time, at a high reading speed, andin fast continuous shooting (or at a high frame rate) in order torestrict blur or focal plane distortion of a moving object and toimprove the object recognition rate.

Here, focal plane distortion is a phenomenon which occurs in an imagingdevice configured to perform a reading operation in units of row as in acomplementary metal oxide semiconductor (CMOS) image sensor or the like,and is a phenomenon that a moving object is distorted in one image dueto gradual offset in reading time per row (see Patent Document 2, forexample). If the reading speed of the imaging device is higher, focalplane distortion is automatically eliminated.

Additionally, FIG. 6 illustrates that the vehicle is going straightahead with the vehicle-mounted camera installed such that the eyedirection matches with the front direction of the vehicle by way ofexample. FIG. 6 can illustrate that a region including a vanishing pointis defined as a center region by way of example. In a case where the eyedirection of the vehicle-mounted camera is tilted toward the frontdirection of the vehicle, or when the vehicle turns right or left notgoing straight ahead, or when the vehicle is traveling on a slope(upward slope or downward slope), the arrangement of the center regionchanges (described below).

Further, other characteristic of an image shot by the vehicle-mountedcamera is that luminance per region is non-uniform while the vehicle istraveling during the nighttime (including cloudy weather or rainyweather) or in a dark place, for example.

FIG. 7 illustrates an exemplary image shot by the vehicle-mounted cameraof the vehicle while traveling during the nighttime or in a dark place.In the illustrated example, a road in the center region 601 in a shotimage 600 is irradiated by the headlamps of the vehicle, and thus it hashigh luminance and can be clearly shot. To the contrary, the lightsemitted from the headlamps do not reach the peripheral region 602, andthe peripheral region 602 stays dark. If the center region 601 and theperipheral region 602 are shot under the same exposure condition or shotby the imaging device with the same sensitivity, the peripheral region602 is shot dark. Alternatively, the exposure condition or thesensitivity of the imaging device is adapted to the peripheral region602, the center region 601 is excessively exposed and may be saturatedto be white (not illustrated). Therefore, the present applicants thinkthat the peripheral region 602 needs to be shot by the imaging device ina longer exposure time or at a higher sensitivity than the center region601.

Additionally, an image shot by a general camera is common with an imageshot by the vehicle-mounted camera in that luminance per region isnon-uniform since an illumination is in the angle of view. However, thevehicle-mounted camera is characterized in that the center region 601irradiated by the headlamps has high luminance and the peripheral region602 where the lights emitted from the headlamps do not reach has lowluminance (or each region with high luminance or low luminance isfixed).

In summary, there is required a shooting apparatus or a shooting controlapparatus capable of realizing an operation of shooting the peripheralregion in a shorter (or longer) exposure time, at a higher readingspeed, or at a higher frame rate than the center region depending on adriving situation of the vehicle. This is similarly applicable to animaging device using not CMOS but charge coupled device (CCD).

B-2. Configuration of Shooting Apparatus

FIG. 31 illustrates an exemplary configuration of a camera module 3100applicable as a vehicle-mounted camera. The illustrated camera module3100 includes a camera control part 3110, a shooting lens 3111, animaging device 3120, an image processing part 3130, a phase differencedetection part 3140, a display processing part 3150, a display part3160, an image output part 3170, an image recording control part 3180,and an image recording part 3190.

The camera control part 3110 controls the entire camera module 3100. Forexample, the camera control part 3110 outputs a control signal to theimaging device 3200 via a signal line 3119, and causes the imagingdevice 3200 to shoot an image in response to a user's operation. Thecontrol signal includes a signal indicating the live view mode or thecapture mode. The live view mode is a mode for shooting an image at acertain interval (per 1/30 seconds, for example) and displaying it onthe display part 3160. On the other hand, the capture mode is a mode forshooting and recording a moving picture or a still image. A movingpicture includes a plurality of images shot at a certain interval. Animage shot in the live view mode is set at a lower resolution than animage shot in the capture mode. Further, the camera control part 3110receives a phase difference detected by the phase difference detectionpart 3140 and controls the positions of the focusing lens and the likein the shooting lens 3111 depending on the phase difference thereby toadjust a focal distance in response to a user's operation.

The shooting lens 3111 is capable of changing a focal distance. Forexample, what is called a four-group zoom lens including a focusinglens, a variator, a compensator, and a master lens (none of which isillustrated) is used as the shooting lens 3111.

The imaging device 3120 converts the amount of light received via theshooting lens 3111 into a potential, and outputs a pixel value dependingon the potential. For example, the imaging device 3120 includes aplurality of normal pixels and a plurality of phase difference pixels. Aphase difference pixel is directed for detecting a phase difference.Each phase difference pixel is configured of a pair of pixels (whichwill be denoted as “left pixel” and “right pixel” below) configured toreceive a pair of eyes-divided lights, respectively. On the other hand,the normal pixels are other than the phase difference pixels, and areused to generate an image. The imaging device 3120 reads the pixelsvalues of the normal pixels and outputs them to the image processingpart 3130 via a signal line 3129 under control of the camera controlpart 3110. Further, the imaging device 3120 reads the pixel values ofthe phase difference pixels and outputs them to the phase differencedetection part 3140 via the signal line 3129.

The image processing part 3130 performs an image processing such asmosaic processing on an image generated by the pixel values of thenormal pixels. The image processing part 3130 holds the image configuredof the pixel values of the normal pixels, interpolates the pixel valuesof the phase difference pixels in the image, further performs an imageprocessing such as mosaic processing or white balance processing on theinterpolated image as needed, and then outputs the processed image tothe display processing part 3150 and the image recording control part3180 via a signal line 3139. Further, the image processing part 3130 mayperform a recognition processing on the shot image.

The phase difference detection part 3140 detects a phase difference fromthe pixel values of the phase difference pixels. For example, the phasedifference detection part 3140 generates distributions of luminance ofleft pixels and right pixels, respectively, and detects a phasedifference from a degree of correlation therebetween. The phasedifference detection part 3140 outputs the detected phase difference tothe camera control part 3110 via a signal line 3149.

The display processing part 3150 performs a display processing such as γcorrection processing, color correction processing, or contrastadjustment processing on the image as needed. The display processingpart 3150 outputs the image subjected to the display processing to thedisplay part 3160 and the image output part 3170 via a signal line 3159.

The display part 3160 displays the image from the display processingpart 3150. Further, the image output part 3170 outputs the image fromthe display processing part 3150 to a device externally connected to thecamera module 3100.

The image recording control part 3180 outputs the image from the imageprocessing part 3130 to the image recording part 3190 via a signal line3189, and causes the image recording part 3190 to record the image. Theimage recording part 3190 records the image passed from the imagerecording control part 3180.

FIG. 32 illustrates an exemplary configuration of the imaging device3120. The imaging device 3120 includes a timing control circuit 3210, arow scanning circuit 3220, a transfer signal generation circuit 3230, apixel array part 3240, a D/A conversion part (DAC) 3250, an A/D (ADC)conversion part 3260, a counter 3270, and a column scanning circuit3290.

The timing control circuit 3210 controls a timing to output a pixelvalue in response to a control signal from the camera control part 3110.The timing control circuit 3210 outputs timing signals Tc and Tr therebyto control the timings to scan the rows and columns. The timing signalTc is directed for indicating a timing to start scanning the rows. Onthe other hand, the timing signal Tr is directed for indicating a timingto start scanning the columns in each row. Here, a row is an arrangementof a plurality of pixels in one direction in the pixel array part 3240,and is also denoted as horizontal line. A row including the phasedifference pixels among the rows (horizontal lines) is denoted as phasedifference line, and a row not including a phase difference pixel isdenoted as normal line. On the other hand, a column is an arrangement ofa plurality of pixels in a direction orthogonal to the rows in the pixelarray part 3240, and is also denoted as vertical line.

Specifically, the timing control circuit 3210 generates a timing signalTc when a shooting period for shooting one image starts, and supplies itto the row scanning circuit 3220 and the transfer signal generationcircuit 3230. The shooting period is divided into a normal pixel outputperiod for outputting the pixel values of the normal pixels, and a phasedifference pixel output period for outputting the pixel values of thephase difference pixels. The timing control circuit 3210 outputs atiming signal Tc when the shooting period starts, and then outputs atiming signal Tc when the phase difference pixel output period starts.The timing control circuit 3210 then generates a timing signal Tr andsupplies it to the column scanning circuit 3290 in synchronization witha timing to select the rows within the shooting period. However, asmaller number of rows are selected in the live view mode, and thus thetiming control circuit 3210 generates a smaller number of timing signalsTr within the shooting period than in the capture mode.

For example, in a case where one image of n rows by m columns includingk phase difference lines is shot, the timing control circuit 3210generates a timing signal Tc once when the shooting period starts, andgenerates a timing signal Tr within the normal pixel output period ntimes. Here, n and m are an integer of 2 or more, and k is an integerbetween 1 and n. The timing control circuit 3210 then generates a timingsignal Tc once when the phase difference pixel output period starts, andgenerates a timing signal Tr within the phase difference pixel outputperiod k times. Further, the timing control circuit 3210 supplies adigital signal indicating a reference voltage value to the D/Aconversion part 3250. Further, the timing control circuit 3210 controlsthe counter 3270 and sets a counter value at the initial value insynchronization with the timings to generate a timing signal Tr.

The row scanning circuit 3220 selects each of the rows according to thetiming signal Tc and the control signal. The row scanning circuit 3220sequentially outputs a row selection signal to each of the rows viasignal lines 3229-1 to 3229-n within the normal pixel output periodthereby to select a row. The row selection signals are set at high levelin a case where a row is selected, and is set at low level in a casewhere it is not selected, for example. Further, the row scanning circuit3220 sequentially selects each of the phase difference lines within thephase difference pixel output period. However, the row scanning circuit3220 selects a smaller number of rows within the shooting period in thelive view mode than in the capture mode. Additionally, the row scanningcircuit 3220 is an exemplary row scanning part described in CLAIMS.

The transfer signal generation circuit 3230 outputs a transfer signal toeach of the pixels in the selected row according to the timing signal Tcand the control signal thereby to drive the pixel. The transfer signalis set at high level in a case where a pixel is driven, and is set atlow level in a case where it is not driven, for example. The transfersignal generation circuit 3230 acquires a timing when the row scanningcircuit 3220 selects a row from the timing signal Tc. The transfersignal generation circuit 3230 drives the respective normal pixels inthe selected row at the same time in synchronization with the rowselection timing within the normal pixel output period. The transfersignal generation circuit 3230 then drives the respective phasedifference pixels in the selected row at the same time insynchronization with the row selection timing within the phasedifference pixel output period. However, a smaller number of rows areselected in the live view mode than in the capture mode, and thus thenormal pixel output period and the phase difference pixel output periodare shorter. Additionally, the transfer signal generation circuit 3230is an exemplary drive part described in CLAIMS.

The pixel array part (pixel region) 3240 is configured in which aplurality of phase difference pixels 3241 and a plurality of normalpixels 3242 are two-dimensionally arranged in a grid shape, for example.Each of the pixels outputs a pixel signal as an electric signal at apotential depending on the amount of received light to the A/Dconversion part 3260 via a signal line of a corresponding column amongsignal lines 3249-1 to 3249-m in a case where it has a high-level rowselection signal and a high-level transfer signal input.

The D/A conversion part 3250 digital to analog (D/A) converts thereference voltage value from the timing control circuit 210, andsupplies a reference voltage Vref to the A/D conversion part 260.

The A/D conversion part 3260 converts an analog pixel signal into adigital signal. The A/D conversion part 3260 includes a plurality of (m,for example) A/D conversion circuits. Each of the A/D conversioncircuits includes a comparator 3262 and a memory 3263. The comparator3262 is directed to comparing the referring voltage Vref with a voltageof a pixel signal and to outputting a comparison result. Each A/Dconversion circuit integrates a pixel signal by an integration circuit(not illustrated), for example, and causes the counter 3270 to count theperiod until the output value of the comparator 3262 indicates that theintegrated voltage exceeds the reference voltage Vref. The value countedby the counter 3270 is then held as a pixel value in the memory 3263.

The memories 3263 are directed to holding pixel values. Each memory 3263has a column selection signal input via a signal line of a correspondingcolumn among signal lines 3298-1 to 3298-m. A column selection signal isdirected to selecting a memory 3263 corresponding to a column and tocausing it to output a pixel value. For example, the column selectionsignal is set at high level in a case where a pixel value is to beoutput, and is set at low level in a case where it is not to be output.The memory 3263 outputs a pixel value via a signal line 3209 in a casewhere the column selection signal is at high level.

The column scanning circuit 3290 reads and outputs a pixel value of eachof the pixels in the selected row according to the timing signal Tr andthe control signal. The transfer signal generation circuit 3230 readsand outputs the pixel values of the normal pixels held in the A/Dconversion part 3260 in a predetermined order whenever it has a timingsignal Tr input within the normal pixel output period. Further, thetransfer signal generation circuit 3230 reads and outputs the pixelvalues of the phase difference pixels held in the A/D conversion part3260 in a predetermined order whenever it has a timing signal Tr input.Here, the column scanning circuit 3290 counts the number of times of thetiming signal Tr thereby to acquire the start and end points of each ofthe normal pixel output period and the phase difference pixel outputperiod. For example, the normal pixel output period is a period afterthe first timing signal Tr is input and until the n-th timing signal Tris input during image shooting for n rows. However, a smaller number ofrows are selected in the live view mode than in the capture mode, andthus the number of rows for the timing signals counted in each period isalso smaller. Additionally, the column scanning circuit 3290 is anexemplary column scanning part described in CLAIMS.

FIG. 8 illustrates an exemplary configuration of a pixel region 800 ofan imaging device applicable as a vehicle-mounted camera according tothe present embodiment. The illustrated pixel region 800 is divided intoa plurality of regions, pixels in a different size are arranged in eachregion, and a sensitivity or a resolution per region is optimizeddepending on an application (or a driving situation during shooting).

In the example illustrated in FIG. 8, the pixel region 800 of theimaging device is divided into three regions including a center region801 and peripheral regions 803 and 802 on the right and left of thecenter region 801, respectively. As described below, region division isrealized by independent control of pixels or pixel region such as the ADconversion processing per region or independent reading in units ofpixel. The peripheral regions 802 and 803 are higher in sensitivity thanthe center region 801 so that the exposure time can be shortened andconsequently the frame rate can be increased, thereby eliminating bluror focal plane distortion in the peripheral regions 802 and 803.

A method for realizing high sensitivity in each divided region may be amethod using gain control (largely increasing a gain in a region inwhich the sensitivity is to be increased) or pixel addition or a methodby adjusting a pixel size (increasing a pixel size of a region in whichthe sensitivity is to be increased). For example, gain control or pixeladdition is performed on part of a region thereby to increase thesensitivity of the region. Further, adjusting a pixel size employs amethod for manufacturing a dedicated imaging device mounting pixels in adifferent size per region. For example, a region in which large-sizepixels are arranged has a higher sensitivity than a region in whichsmall-size pixels are arranged. Additionally, a region on which pixeladdition is performed can be assumed to adjust an apparent pixel size tobe larger.

In the example illustrated in FIG. 8, fine pixels are arranged in thecenter region 801, while large-size pixels are arranged in each of theperipheral regions 802 and 803. However, the following descriptionassumes that pixel addition is performed in the peripheral regions 802and 803 both in an imaging device manufactured with a pixel size changedper divided region and in an imaging device in which uniform-size pixelsare arranged over the pixel region. Further, FIG. 8 illustrates a pixelarrangement of only part of each of the regions 801 to 83 for simpleillustration. Additionally, the arrangement of uniform-size pixels overthe pixel region includes a case where pixels in different sizes aremixed but pixels in each size are uniformly distributed in the pixelregion as illustrated in FIG. 34 or FIG. 35 in addition to a case whereall the pixels are in the same size as illustrated in FIG. 33.

The center region 801 is configured of fine pixels, and thus realizeshigh-resolution shooting. On the other hand, the peripheral regions 802and 803 are configured of large-size pixels and thus have a lowresolution. It is not an object to shoot the peripheral regions 802 and803 at a low resolution, but the large-size pixels have a large lightreceiving area, thereby realizing high sensitivity. Thus, due to highsensitivity, the exposure time of the peripheral regions 802 and 803 canbe shortened and blur can be reduced when fast driving or shooting amoving object. Further, the exposure time is increased so that theperipheral regions 802 and 803 (which are not irradiated by theheadlamps) can be shot at a sufficient sensitivity while travelingduring the nighttime or in a dark place.

Further, when the pixel size is increased and the resolution is lowered,the number of pixels per row (or per unit length) is reduced, and thusthe reading speed of the peripheral regions 802 and 803 improves(assuming that the pixel rate (reading time per pixel) is constant).Therefore, the peripheral regions 802 and 803 can be shot at a highframe rate (or can be continuously shot at a high speed), and theperipheral regions 802 and 803 are continuously shot at a high speedwhen fast driving or shooting a moving object, thereby improving theobject recognition rate of the moving object.

If the imaging device in which pixels in different sizes are arrangedbetween the center region and the peripheral regions as illustrated inFIG. 8 is employed, the shooting conditions (exposure condition,sensitivity, frame rate, and reading speed) can be controlled perregion. A method for configuring an imaging device with a differentpixel size per region may be a method for apparently increasing a pixelsize of part of a region by a signal processing such as pixel additionor gain adjustment in an imaging device in which uniform-size pixels arearranged, or a method for manufacturing an imaging device in whichpixels in a different size are arranged per region. In consideration ofdesign or productive efficiency of pixels or on-chip lenses, the formerconfiguration method using an imaging device in which uniform-sizepixels are arranged is more advantageous. Further, the former method isadvantageous in that the position and the size of each of the centerregion and the peripheral regions can be adaptively changed by a signalprocessing. Additionally, the technology for the imaging device in whichpixels in different sizes are arranged is disclosed also in PatentDocument 3 or Patent Document 4, for example.

Further, the present embodiment assumes that the pixel region 800 of theimaging device is scanned (pixels are read) per divided region inparallel. FIG. 9 illustrates how the pixel region 800 is scanned perdivided region. In the Figure, scan lines in the respective regionsincluding the center region 801 and the peripheral regions 802 and 803are denoted with reference numerals 901, 902, and 903, respectively. Thecenter region 801 is different in pixel size from the peripheral regions802 and 803, and an interval between the scan lines is differenttherebetween. That is, the number of scan lines in the peripheralregions 802 and 803 is smaller and the reading time per line is shorterthan in the center region 801, and thus a time to read one frame can beshorter in the peripheral regions 802 and 803, thereby achieving ahigher frame rate.

FIG. 10 schematically illustrates an exemplary configuration of a cameramodule 1000 configured to read each of the regions 801 to 803 in thepixel region 800 of the imaging device in parallel. In the Figure, afirst reading and AD conversion part 1001 reads a pixel in the centerregion 801 and AD-converts a reading signal. Further, a second readingand AD conversion part 1002 and a third reading and AD conversion part1003 read a pixel in the peripheral regions 802 and 803 and AD-convert areading signal in parallel with the first reading and AD conversion part1001. Each of the reading and reading signal AD conversion parts 1001 to1003 outputs the AD-converted pixel signal to an image processing part1004. The shooting apparatus can be configured in which the pixel arrayin which pixels in different sizes are arranged in the center region 801and the peripheral regions 802 and 803 is assumed as the first layer,the reading signal AD conversion parts 1001 to 1003 in the regions 801to 803 are assumed as the second layer, and the image processing part1004 at the final stage is assumed as the third layer and the layers arelaminated. The image processing part 1004 performs the objectrecognition processing or the like on a moving object captured in eachof the regions 801 to 803, for example. The pixel region 800 is scannedper divided region in parallel and is subjected to a signal processingsuch as AD conversion, thereby individually controlling the exposuretime or the frame rate per region. Additionally, the pixel region 8000and the AD conversion parts 1001 to 1003 are generally combined (orincluding a memory region configured to accumulate AD-converted pixelsignals as needed) thereby to configure a single imaging device, and theimage processing part 1004 at the latter stage is added to the imagingdevice thereby to configure the shooting apparatus (camera module).However, in the case of a vehicle-mounted camera, the functioncorresponding to the image processing part 1004 is not mounted as acircuit module in the camera module, but may be integrated in a controlcircuit chip of the vehicle.

Assuming that the pixel rate is constant, the center region 801 has asmall pixel size and a high resolution, but the reading speed of thereading signal AD conversion part 1001 is lower as the number of pixelsper line is larger, and thus the frame rate in the center region 801 islower. On the other hand, the peripheral regions 802 and 803 have alarge pixel size, and thus have a high sensitivity and a low resolution,but if the respective reading and reading signal AD conversion parts1001 to 1003 are assumed as circuits operating at the same operationfrequency in the same chip, a higher frame rate of the peripheralregions 802 and 803 can be realized.

Additionally, the imaging device, which is configured such that a signalprocessing such as AD conversion per region can be performed inparallel, is disclosed also in Patent Document 5 or Patent Document 6,for example.

FIG. 11 schematically illustrates a shooting range 1100 when theshooting apparatus having the configuration illustrated in FIG. 8 toFIG. 10 is installed near the center of the front nose of the vehiclesuch that the eye direction matches with the traveling direction of thevehicle. The shooting apparatus divides and shoots the shooting range1100 into three shooting ranges 1101 to 1103, and performs an imageprocessing.

The shooting range denoted with the reference numeral 1101 is a regionof gaze which is shot at a high resolution in the center region 801 inthe pixel region 800. For example, an object, which enters the region ofgaze, such as the back of a leading vehicle, a road sign (road guidance)installed along a road, a road sign drawn on a road, the tail lamps of aleading vehicle, or a pedestrian walking on a crosswalk, can be shot ata high resolution in the center region 801. Further, the imageprocessing part 1004 can accurately recognize the object from the imageshot in the center region 801, or can measure the distance to theobject.

On the other hand, the shooting ranges denoted with the referencenumerals 1102 and 1103 are fast-moving object recognition emphasizedregions shot in the peripheral regions 802 and 803 in the pixel region800. As previously described with reference to FIG. 3, while the vehicleis driving at a high speed, an object coming into the shooting range1102 or 1103 moves faster than an object in the shooting range 1101. Theperipheral regions 802 and 803 are configured of large-size pixels andhave a low resolution, but have a high reading speed, thereby shootingat a high frame rate (or continuously shooting at a high speed) andimproving a fast-moving object recognition accuracy. Further, theperipheral regions 802 and 803 have a large pixel size and a highsensitivity, and thus the exposure time can be further shortened andblur can be further reduced than in the region of gaze 1101.

Additionally, the fast-moving object recognition emphasized regions arenot shot in the peripheral regions 802 and 803 with a low resolution anda high sensitivity in the pixel region 800, but an edge processing maybe applied to images shot in the peripheral regions 802 and 803 with astill high resolution (in the development process, for example), therebyenhancing the object recognition rate.

B-3. Shooting Condition Processing Per Region

The shooting apparatus illustrated in FIG. 8 to FIG. 10 is configured toscan the pixel region 800 per divided region in parallel and to performa signal processing, and can individually control the shootingconditions such as exposure time, sensitivity, frame rate, and readingspeed per region (as described above).

FIG. 12 illustrates an exemplary timing chart on the exposure/readingprocessing per region in the pixel region 800. As described above, theexposure/reading processing in the center region 801 and the peripheralregions 802 and 803 is performed in parallel. The horizontal axis is atime axis and the vertical axis is a scan line (row number) in theFigure. Here, the imaging device assumes an imaging device configured toperform the reading operation per row such as CMOS image sensor.

The parallelograms in gray in FIG. 12 indicate an exposure operation ineach of the center region 801 and the peripheral regions 802 and 803.Here, a time T1 corresponding to a length of the base of a parallelogramis an exposure time for one scan line with reference to FIG. 13illustrating the timing chart of the exposure processing for one frame.The exposure time is basically determined by the shutter speed. Further,a time T2 corresponding to an interval between adjacent parallelogramsis a frame interval, and a value obtained by dividing a unit time by T2indicates a frame rate. Further, an angle θ formed between an obliqueside and the base of a parallelogram corresponds to an offset in readingtime per scan line, or a reading speed. When the reading speed is low,the angle θ is small, and focal plane distortion (described above)easily occurs when shooting a moving object. To the contrary, when thereading speed is high, the angle θ is large, and focal plane distortioncan be restricted. At a lower resolution (or in a smaller number ofpixels per line), the reading speed is higher. The reading speed ishigher also at a higher AD conversion speed.

Referring to FIG. 12 again, it is assumed that the center region 801 isconfigured of small-size pixels and a high-resolution image is shottherein. On the other hand, the peripheral regions 802 and 803 areconfigured of large-size pixels and have a high sensitivity, and thuscan have a shorter exposure time than the center region 801 (the length(T1) of the base of a parallelogram of the peripheral regions 802 and803 is shorter in FIG. 12). Thus, it is possible to reduce bluroccurring in the landscapes in the peripheral regions when shooting amoving object or fast driving. Further, a higher frame rate may beachieved by use of the short exposure time of the peripheral regions 802and 803 (the interval T2 between adjacent parallelograms can be reducedin FIG. 13), thereby improving the object recognition accuracy when fastdriving or shooting a moving object.

Further, the peripheral regions 802 and 803 are set at a lowerresolution, and thus can be higher in the reading speed than the centerregion 801. Therefore, the tilt θ of oblique lines of a parallelogram ofthe peripheral regions 802 and 803 is larger, thereby restricting focalplane distortion. Of course, the reading speed is higher also byincreasing the signal processing speed (such as AD conversion speed) inthe peripheral regions, not setting the peripheral regions at a lowerresolution, thereby restricting blur or focal plane distortion.

Essentially, the peripheral regions 802 and 803 can be set at a higherframe rate and accurate object recognition can be performed therein bythe exposure/reading operation as illustrated in FIG. 12, and blur in afast-moving object can be reduced due to the short exposure time. Theexposure/reading operation illustrated in FIG. 12 is effective when thevehicle is driving at a high speed or a moving object is recognized, forexample.

For example, in a case where the technology disclosed in the presentspecification is applied to the shooting part 2916 configured to shootbehind the vehicle when the vehicle changes lanes for passing whiletraveling on an expressway, the lanes shot in the peripheral regions oran approaching vehicle is accurately recognized, thereby performingaccurate passing control.

FIG. 14 illustrates other exemplary timing chart on the exposure/readingprocessing in the pixel region 800. The processing is common with thereading processing illustrated in FIG. 12 in that the peripheral regions802 and 803 are shot in a shorter exposure time than the center region801, but is different therefrom in that the peripheral regions 802 and803 are set at the same frame rate as the center region 801 (or theperipheral regions 802 and 803 are horizontally flowed at a high speed).Fast reading is performed in the peripheral regions 802 and 803, blur orfocal plane distortion in a moving object is reduced, and thus afast-moving object can be accurately recognized even not at a high framerate.

Further, FIG. 15 illustrates still other exemplary timing chart on theexposure/reading processing in the pixel region 800. In the illustratedexample, the peripheral regions 802 and 803 and the center region 801are set at the same exposure time and at the same frame rate forshooting. Large-size pixels are arranged in the peripheral regions 802and 803, and the sensitivity therein is higher for the light receivingarea even in the same exposure time (described above). Therefore, theexposure time of the peripheral regions 802 and 803 is set as long asthat of the center region 801, and thus visibility of a low-illuminanceobject is improved. For example, low-illuminance objects, which are notirradiated by the headlamps, are shot in the peripheral regions 802 and803 while traveling during the nighttime (see FIG. 7, for example), butthe low-illuminance objects can be captured at a high sensitivity by theexposure processing illustrated in FIG. 15. Though not illustrated, theexposure time of the peripheral regions 802 and 803 may be longer thanthat of the center region 801. When the images in the respective regions801 to 803 shot under different exposure conditions are read andcombined into one image, gray level correction is made thereby toacquire the entire image with high visibility.

Additionally, a change in the image per frame in the center region issmall as described with reference to FIG. 3. Thus, a plurality of framesin the center region may be combined and displayed. For example, aplurality of frames of the center region shot while changing exposureare high dynamic range (HDR) combined thereby to generate an image witha wide dynamic range.

B-4. Shooting Condition Control Depending on Driving Situation

With the shooting apparatus including the imaging device in which pixelsizes are different between the center region and the peripheral regionsas illustrated in FIG. 8 to FIG. 10, the shooting conditions of theperipheral regions 802 and 803 are adaptively controlled relative to thecenter region 801 depending on a driving situation of the vehicle,thereby realizing a reduction in blur, the accurate recognitionprocessing on a moving object, high-sensitivity shooting of alow-illuminance object, and the like. The adjustment parameters of theshooting conditions of the peripheral regions 802 and 803 may beexposure time, sensitivity, frame rate, resolution, and reading speed.The appropriate shooting conditions of the peripheral regions 802 and803 in each driving situation are indicated per adjustment parameter inthe following Table 1. However, the Table indicates that differentialadjustment of the peripheral regions 802 and 803 is made relative to thecenter region 801 by way of example, and indicates “high” or “low”relative to the center region 801. Further, the adjustment parameterswhich do not need the differential adjustment (or which may requirenormal adjustment) are denoted with “−”. Additionally, the definitionsof “exposure time” and “frame rate” in the Table are as described above.

TABLE 1 Shooting conditions of peripheral regions Exposure Frame ReadingDriving situation time Sensitivity rate speed During stop/normal — — — —traveling During fast traveling Short — High High Going-through/downtownShort High — — Nighttime, dark place Long High — — Moving object ShortHigh High High recognition Abnormal driving — — High High Traveling intunnel — High High —

Effects by adjusting each adjustment parameter and an adjustment methodwill be first described. The exposure time can be basically adjusted bythe shutter speed of the shooting apparatus. When the exposure time isshortened, the shutter speed is increased (as described above), therebyrestricting blur. Thus, the exposure time is shortened, thereby reducingblur occurring in the landscapes in the peripheral regions when shootinga moving object or fast driving. The amount of received light of theimaging device is small in a short exposure time, and thus thesensitivity lowers or the sensitivity needs to be increased. To thecontrary, there is an effect that the amount of received light of theimaging device increases and the sensitivity is high in a long exposuretime.

When the sensitivity is increased, an object can be shot in a dark placesuch as during the nighttime or in a tunnel (or in the peripheralregions where the lights emitted from the headlamps do not reach).Further, shooting in a short exposure time is enabled at a highsensitivity, and consequently there can be derived an effect that bluroccurring in the landscapes in the peripheral regions is reduced in ashort exposure time when shooting a moving object or fast driving. Forexample, the gain in the shooting signal processing is increased therebyto increase the sensitivity. Further, when the pixel size is increased,the amount of received light per pixel increases and a high sensitivityis achieved. With the imaging device equipped with the pixel additionfunction, the apparent pixel size increases by addition, and thesensitivity similarly increases. However, when the pixel size isincreased or pixel addition is performed, the resolution lowers.

When the frame rate is increased, the number of frames processed perunit time increases, and thus a motion of a moving object can besmoothly captured, thereby reducing blur occurring in the landscapes inthe peripheral regions while fast driving. The shooting signalprocessing speed has to be increased in order to increase the framerate, but a high frame rate can be realized by increasing the ADconversion speed or the circuit operation clock frequency. Further, theexposure time needs to be short in order to increase the frame rate, andthe sensitivity lowers or the sensitivity needs to be increased. Amethod for improving a gain, increasing a pixel size, performing pixeladdition or thinning reading, or the like is employed in order toincrease the sensitivity. However, when pixel addition or thinningreading is performed, the resolution lowers. When pixel addition isperformed, the sensitivity increases, but the sensitivity does notimprove by thinning reading.

When the reading speed is increased, the reading time is shortened (orthe angle θ in FIG. 13 increases), thereby reducing focal planedistortion when shooting a moving object. The shooting signal processingspeed is increased (the AD conversion speed is improved by largelyincreasing the circuit operation clock frequency, for example), therebyincreasing the reading speed.

During normal (or low-speed) traveling or backward traveling, the framerate and the exposure time are the same between the center region andthe peripheral regions (see FIG. 15, for example), and an image isuniform between the center region and the peripheral regions. Forexample, when the drive system control unit 2100 recognizes normaltraveling or backward traveling on the basis of the rotation speed ofthe wheels detected by the vehicle state detection part 2110, thevehicle exterior information detection unit 2400 may designate the framerate and the exposure time of the peripheral regions for the shootingpart 2410.

During fast traveling, the peripheral regions are set at a shorterexposure time, a higher frame rate, and a higher reading speed than thecenter region (see FIG. 12, for example). The center region is set at anormal frame rate and a normal exposure time with the resolutionemphasized (or still at a high resolution). On the other hand, theexposure time is shortened in the peripheral regions, thereby reducingblur occurring in the landscapes in the peripheral regions when fastdriving. Additionally, it may be preferable that the peripheral regionsare set at a high sensitivity in order to compensate for the exposure ina short exposure time. Further, the peripheral regions are set at a highframe rate, thereby reducing blur occurring in the landscapes in theperipheral regions. Further, the peripheral regions are set at a highreading speed, thereby reducing focal plane distortion occurring in thelandscapes in the peripheral regions while fast driving. The frame rateis increased, the exposure time is shortened, and the resolution of amoving object is emphasized. For example, when the drive system controlunit 2100 recognizes fast traveling on the basis of the rotation speedof the wheels detected by the vehicle state detection part 2110, thevehicle exterior information detection unit 2400 may designate at leastone of the shorter exposure time, the higher frame rate, and the higherreading speed of the peripheral regions for the shooting part 2410.

While going through or traveling downtown, an object approaching by thevehicle needs to be sensed. Thus, while going through or travelingdowntown, the peripheral regions are set at a shorter exposure time anda higher sensitivity than the center region. The exposure time isshortened, and thus an object approaching by the vehicle can be capturedwith less blur. Further, the sensitivity is increased, and thus shootingis enabled in a short exposure time. For example, whether the street isnarrow or whether a building is approaching is sensed on the basis ofmap information or road information acquired in the navigation systemincluded in the instrument panel 2730, and the vehicle exteriorinformation detection unit 2400 may designate the shorter exposure timeand the higher sensitivity of the peripheral regions for the shootingpart 2410.

However, it is enough that the exposure time is shortened and thesensitivity is increased only in a peripheral region where an object isapproaching, not in both regions, while going through or travelingdowntown (a peripheral region where a moving object to be recognized isnot present does not need to be adjusted). For example, which side ofthe vehicle an object is approaching is recognized on the basis of arecognized object in an image shot by the shooting part 2410, or thesurrounding information detection sensor included in the vehicleexterior information detection part 2420, and the vehicle exteriorinformation detection unit 2400 may designate the frame rate and theexposure time of the peripheral region for the shooting part 2410.

It is assumed that while traveling during the nighttime or in a darkplace (such as in a tunnel), the headlamps are lit to illuminate thecenter region but the lights emitted from the headlamps do not reach theperipheral regions. Thus, while traveling during the nighttime or in adark place (such as in a tunnel), assuming that both the peripheralregions 802 and 803 and the center region 801 are adjusted at a highsensitivity, the peripheral regions 802 and 803 are further adjusted ata longer exposure time and a higher sensitivity. Since when the exposuretime is increased, the amount of received light of the imaging deviceincreases, the peripheral regions where the lights emitted from theheadlamps do not reach can be shot at a high sensitivity. Further, theperipheral regions are configured of an imaging device in a large pixelsize or are subjected to pixel addition, thereby shooting at a lowresolution but at a high sensitivity. For example, the vehicle exteriorinformation detection unit 2400 may output an instruction in response toan input operation to the shooting part 2410 when the driver adjusts thesensitivity to be higher via the input part 2800 during the nighttime orin a dark place. Alternatively, when the nighttime or a dark place (orreduced illuminance) is sensed on the basis of a detection result of thesunshine sensor included in the vehicle exterior information detectionpart 2420, the vehicle exterior information detection unit 2400 mayinstruct the shooting part 2410 to increase the exposure time in theperipheral regions. Further, when an entry into a tunnel is sensed onthe basis of map information or road information acquired in thenavigation system, the vehicle exterior information detection unit 2400may designate the exposure time of the peripheral regions for theshooting part 2410.

Alternatively, the headlamps are lit during the nighttime (includingcloudy weather or rainy weather) or in a dark place, and thus the bodysystem control unit 2200 may designate the exposure time of theperipheral regions for the shooting part 2410 in response to the litheadlamps. As previously described with reference to FIG. 7, the centerregion irradiated by the headlamps is at high luminance and can beclearly shot, but the lights emitted from the headlamps do not reach theperipheral regions, and thus the peripheral regions are shot at a highersensitivity and in a longer exposure time. Further, the exposure time ofthe peripheral regions may be designated for the shooting part 2410 whenthe body system control unit 2200 switches the headlamps to high beam orlow beam.

The peripheral regions are set at a shorter exposure time, a highersensitivity, a higher frame rate, and a higher reading speed than thecenter region when recognizing a moving object. The moving objectrecognition rate lowers also in the center region, but the recognitionrate further lowers in the peripheral regions. The exposure time isreduced, thereby reducing blur occurring in a moving object. Further,when the sensitivity is increased, shooting is enabled in a shorterexposure time, thereby reducing blur occurring in a moving object.Further, the frame rate is increased, thereby reducing blur occurring ina moving object. Furthermore, the reading speed is increased, therebyreducing focal plane distortion in a moving object.

Abnormal driving is spinning, slipping, lateral turning, and the like.Generally, spinning is that the tires slip on a road, the vehicle bodyrotates, and a target direction is largely different from theorientation of the vehicle, and slipping is that the tires slip but thevehicle body does not largely rotate. Similarly as in fast traveling,the frame rate is increased and the exposure time is shortened in theperipheral regions during abnormal driving thereby to emphasize theresolution of a moving object. For example, the acceleration sensorincluded in the vehicle state detection part 2110 detects addition ofabnormal acceleration to the vehicle, or recognizes an abnormal motionof a surrounding image shot by the shooting part 2410, thereby sensingabnormal driving. Then, the vehicle exterior information detection unit2400 may designate the frame rate and the exposure time of theperipheral regions for the shooting part 2410.

It is assumed that while traveling in a tunnel, the headlamps are lit toilluminate the center region but the lights emitted from the headlampsdo not reach the peripheral regions. Thus, while traveling in a tunnel,the peripheral regions are adjusted to have a higher sensitivity and ahigher frame rate than the center region. The sensitivity is increased,thereby preferably shooting the peripheral regions where the lightsemitted from the headlamps do not reach. Further, the frame rate isincreased, thereby reducing blur occurring in the landscapes (the wallsof the tunnel) in the peripheral regions. For example, when traveling ina tunnel is determined on the basis of information of the carnavigation, the vehicle exterior information detection unit 2400 mayinstruct the shooting part 2410 to adjust the peripheral regions to havea higher sensitivity and a higher frame rate. Alternatively, travelingin a tunnel can be recognized on the basis of a recognition result of animage shot by the vehicle-mounted camera.

Additionally, though omitted in Table 1, resolution may be included inthe adjustment parameters of the shooting conditions. For example, in acase where an imaging device (described below) capable of increasing theapparent pixel size by pixel addition or thinning reading is used, theresolution can be adjusted. For example, a region where pixel additionis performed has a lower resolution but a higher sensitivity, therebyshortening the exposure time and further increasing the frame rate.

Development mode may be further included in the adjustment parameters ofthe shooting conditions. For example, the center region is set in adevelopment processing mode with color reproducibility and visibilityemphasized, while the peripheral regions are set in a developmentprocessing mode of performing simple development or edge emphasis inorder to improve visibility of a moving object.

FIG. 29 is a flowchart illustrating a processing procedure of performingshooting condition control of each region (peripheral region) dependingon a driving situation on an imaging device provided with a centerregion and peripheral regions in a pixel region. The integrated controlunit 2600 in the vehicle control system 2000 mainly executespredetermined programs, for example, so that the illustrated processingprocedure is realized.

At first, a current driving situation (fast traveling,going-through/traveling downtown, traveling during the nighttime or in adark place, appearance of a moving object, abnormal driving, travelingin a tunnel, or the like) of the vehicle is grasped on the basis of adetection result of at least one of the vehicle state detection part2110, the vehicle exterior information detection part 2420, and thevehicle interior state detection part 2510, an analysis result of animage shot by the shooting part 2410, or the like (step S2901).

Then, the shooting conditions (exposure condition, sensitivity, framerate, and reading speed) of the peripheral regions suitable for thedriving situation are determined on the basis of Table 1, for example,(steps S2902 and 2903).

Then, the exposure processing is performed on the vehicle-mounted cameraunder the determined shooting conditions (step S2904).

Further, the recognition processing is performed on an image shot by thevehicle-mounted camera as described above (step S2905), and the vehicledriving control may be performed on the basis of a recognition result ofthe peripheral regions, or the like (step S2906). The driving controlwill be described below in detail.

B-5. Shooting Condition Control Using Imaging Device where Uniform-SizePixels are Arranged

There will be described herein methods for controlling the shootingconditions (exposure time, sensitivity, frame rate, and reading speed)in a case where an imaging device in which uniform-size pixels arearranged is used. The following methods are combined thereby to performcontrol depending on a driving situation described in B-4.

The exposure time corresponds to the shutter speed. The sensitivity ofpixels needs to be improved in order to realize a short exposure time. Amethod for largely increasing a gain or a method for increasing anapparent pixel size by pixel addition may be employed in the imagingdevice in which uniform-size pixels are arranged. However, when pixeladdition is performed, the resolution lowers. For example, the exposuretime is shortened in the peripheral regions, thereby reducing bluroccurring in the landscapes in the peripheral regions when shooting amoving object or fast driving.

Further, a method for largely increasing a gain or a method forincreasing an apparent pixel size by pixel addition may be employed toincrease the sensitivity in the imaging device in which uniform-sizepixels are arranged. However, pixel addition causes a low resolution.When the sensitivity is increased, an object in a dark place such asduring the nighttime or in a tunnel (or in the peripheral regions wherethe lights emitted from the headlamps do not reach) can be clearly shot.Further, if the sensitivity is high, shooting in a short exposure timeis enabled, and there is consequently derived an effect that bluroccurring in the landscapes in the peripheral regions can be reduced inthe short exposure time when shooting a moving object or fast driving.

Further, the shooting signal processing speed has to be improved inorder to increase the frame rate in the imaging device in whichuniform-size pixels are arranged, but a higher frame rate can berealized by increasing the AD conversion speed or the circuit operationclock frequency. Further, the number of apparent pixels is reduced bypixel addition or thinning processing to reduce a processing load perframe, thereby realizing a higher frame rate. Further, the exposure timehas to be short in order to increase the frame rate, and the sensitivitylower or the sensitivity needs to be increased. A method for improving again or performing pixel addition may be employed for increasing thesensitivity, for example. However, when pixel addition is performed, theresolution lowers. Additionally, the processing time per frame can bereduced due to a reduction in the number of pixels by thinning reading,but the sensitivity does not improve. When the frame rate is increased,the number of frames processed per unit time increases, and thus amotion of a moving object can be smoothly captured, thereby reducingblur occurring in the landscapes in the peripheral regions while fastdriving.

Further, the shooting signal processing speed may be increased (forexample, the AD conversion speed is improved by largely increasing thecircuit operation clock frequency, for example) in order to increase thereading speed in the imaging device in which uniform-size pixels arearranged. When the reading speed is increased, the reading time isshortened (or the angle θ in FIG. 13 increases), thereby reducing focalplane distortion when shooting a moving object.

Additionally, pixel addition is directed to acquiring an addition signalof pixel values of a plurality of pixels with the same color in thepixel value reading processing. For example, electrons generated in eachpixel to be added are accumulated in floating diffusion (FD) to beadded, thereby realizing pixel addition. The pixels subjected to pixeladdition are apparently one pixel. That is, the pixels subjected topixel addition are lower in resolution, and are consequently to bepixels capable of being read at a high sensitivity and at a high readingspeed. However, pixels to be subjected to pixel addition are assumed asadjacent pixels, or pixels not adjacent but in a short distance. Withreference to the examples illustrated in FIG. 8 to FIG. 10, pixeladdition is not performed in the center region thereby to keep theresolution high, while pixel addition is performed in the peripheralregions, thereby realizing a high sensitivity and a high frame rate inthe peripheral regions.

For a color layout, the Bayer layout is typical, which is a periodiclayout assuming four pixels in 2×2 as a unit layout, where two pixels inthe four pixels in the unit layout are obliquely arranged as G pixelsand the other two pixels are an R pixel and a B pixel. There is known,in an imaging device in the Bayer layout, a method for performing pixeladdition by same-colored pixel addition/reading for adding and reading aplurality of pixels for which the color filters of the same color areadjacent in the horizontal direction or in the vertical direction, or inthe horizontal direction and in the vertical direction, for example.

However, the color layout is not limited to the Bayer layout and may beother layout pattern, and in this case, pixel addition may be performedby same-colored pixel addition/reading. Here, other layout pattern isnot limited to a periodic layout of unit pixels of 2×2, and may be aperiodic layout of unit pixels of 3×3, 3×4, or the like.

FIG. 22 illustrates an exemplary imaging device configured to performregion division on a pixel region by pixel addition by way of example.Each pixel of RGB is configured in 2×2 pixels, and the number ofeffective pixels is entirely ¼. However, a small number of pixels aredrawn for the simplified Figure, but it should be understood that thepixel array is actually configured of a large number of pixels in thevertical and horizontal directions.

The inside of a region surrounded in a bold line denoted with thereference numeral 221 is a center region in FIG. 22, where pixeladdition is not performed and the pixel value reading processing isperformed. Thus, a shot image in the center region is kept at a highresolution. On the other hand, the peripheral region is outside the boldline 221, where 2×2 pixel addition is performed on each pixel of RGB sothat each of the pixels functions as a pixel apparently in a four-timesize. Therefore, the peripheral region is ¼ times lower in theresolution but is higher in the sensitivity than the center region (orthan in the case of the normal pixel value reading processing withoutpixel addition), and is capable of being fast read. Of course, if pixeladdition is not performed in the imaging device illustrated in FIG. 22,the imaging device can be used as a normal imaging device (pixeladdition is not performed and all pixels are read). For example, theimaging device may be used as an imaging device configured to read allpixels during stop or normal traveling.

FIG. 22 illustrates an exemplary circuit configuration configured to addand then read charges accumulated per 2×2 pixels, but a signal read fromeach pixel may be subjected to the addition processing in the signalprocessing part. Further, FIG. 22 illustrates that region division isperformed on the imaging device in the 2×2 pixel layout by pixeladdition by way of example, but the technology disclosed in the presentspecification is not limited to a specific color filter layout. Forexample, the configuration for pixel addition is employed thereby toperform similar region division as described above also in imagingdevices in various color layouts such as RGBIR using some of G pixels asIR (infrared sensor), RGBW including white pixels, a layout embeddingphase difference pixels therein (such as layout in which some of Rpixels in the 2×2 pixel layout are replaced with phase differencepixels, layout using a polarization filter for some pixels, or thelike), and layout system other than the Bayer layout.

Additionally, pixel addition itself is a technology disclosed also inPatent Document 7, and is used for high-sensitivity shooting, forexample. Further, the reading speed can be increased also by thethinning processing, not by pixel addition, but the sensitivity cannotbe increased.

B-6. Shooting Condition Control Using Dedicated Imaging Device

There will be subsequently described methods for controlling theshooting conditions (exposure time, sensitivity, frame rate, and readingspeed) in a case where a dedicated imaging device in which pixels in adifferent size are arranger per region of a center region and peripheralregions. Unless particularly stated, it is assumed that pixels with asmaller size and a higher resolution are arranged in the center regionthan in the peripheral regions, or pixels in a larger size are arrangedin the peripheral regions than in the center region. The followingmethods are combined thereby to perform control depending on a drivingsituation described in B-4.

As previously described, large-size pixels have a high sensitivity and ahigh reading speed. Due to a high sensitivity, a short exposure time canbe easily realized, and as the exposure time is longer, the sensitivityis much higher. Further, due to a shot exposure time and a high readingspeed, a high frame rate can be easily realized.

The exposure time corresponds to the shutter speed. The regions wherelarge-size pixels are arranged have a high sensitivity and thus theexposure time can be shortened. For example, the peripheral regionswhere large-size pixels are arranged are set at a short exposure time,thereby reducing blur occurring in the landscapes in the peripheralregions when shooting a moving object and fast driving.

Further, since the regions where large-size pixels are arranged have ahigh sensitivity, even if the lights emitted from the headlamps do notreach the regions while traveling during the nighttime or in a darkplace, the regions are set at a long exposure time so that thesensitivity is much higher and an object can be clearly shot. Further,the sensitivity is further increased in the method for largelyincreasing a gain. Of course, pixel addition may be employed together.

Further, the regions where large-size pixels are arranged have a highsensitivity, and thus have a short exposure time and a high readingspeed so that the frame rate can be easily increased. The AD conversionspeed or the circuit operation clock frequency is increased, therebyrealizing a higher frame rate. When the frame rate is increased, thenumber of frames processed per unit time increases, and thus a motion ofa moving object can be smoothly captured, thereby reducing bluroccurring in the landscapes in the peripheral regions when fast driving.

Further, the regions where large-size pixels are arranged originallyhave a high reading speed. The shooting signal processing speed isincreased (the AD conversion speed is improved by largely increasing thecircuit operation clock frequency, for example), thereby furtherincreasing the reading speed. When the reading speed is increased, thereading time is shortened (or the angle θ in FIG. 13 increases), therebyreducing focal plane distortion when shooting a moving object.

B-7. Variation of Region Division

FIG. 8 illustrates that the pixel region 800 of the imaging device isdivided into three regions such that the center region 801 is arrangedalmost at the center and the peripheral regions 803 and 802 are on theright and left sides of it, respectively, by way of example. The regiondivision can be applied in a case where the shooting apparatus isinstalled almost at the center of the front nose of the vehicle suchthat the eye direction directs in the eye direction of the vehicle, forexample, as illustrated in FIG. 11.

An optimum region division method is different depending on a placewhere the shooting apparatus is installed in the vehicle, or anorientation of the eye direction of the imaging device at itsinstallation place (a tilt relative to the traveling direction of thevehicle).

For example, in the case of a shooting apparatus which is installed nearthe left end of the front nose of the vehicle (or near the head of theleft fender or near the left headlamp) such that the eye direction istilted leftward from the traveling direction of the vehicle as denotedwith the reference numeral 1601 in FIG. 16, it is preferable that regiondivision is performed such that a center region 1701 with a highresolution is arranged leftward from the center of the pixel region andperipheral regions 1703 and 1702 with a low resolution (or a highsensitivity and a high reading speed) are arranged on the right and leftsides of it, respectively, as illustrated in FIG. 17. The leftperipheral region 1702 is narrower and the right peripheral region 1703is wider.

Similarly, in the case of a shooting apparatus which is arranged nearthe right end of the front nose of the vehicle (or near the head of theright fender or near the right headlamp) such that the eye direction istilted rightward from the traveling direction of the vehicle as denotedwith the reference numeral 1801 in FIG. 18, it is preferable that regiondivision is performed such that a center region 1901 with a highresolution is arranged rightward from the center of the pixel region andperipheral regions 1903 and 1902 with a low resolution (or a highsensitivity and a high reading speed) are arranged on the right and leftsides of it, respectively, as illustrated in FIG. 19. The leftperipheral region 1902 is wider and the right peripheral region 1903 isnarrower.

Further, FIG. 8 illustrates the pixel region 800 which is configured ofthe high-resolution center region 801 configured of small-size pixelsand the low-resolution (high sensitivity and high reading speed)peripheral regions 802 and 803 configured of large-size pixels by use oftwo kinds of pixels in different pixel sizes by way of example. As avariant, a pixel region 200 divided into three kinds of regions may beconfigured as illustrated in FIG. 20. In the Figure, the referencenumeral 201 denotes a center region configured of small-size pixels, thereference numerals 202 and 203 denote first peripheral regionsconfigured of middle-size pixels, and the reference numerals 204 and 205denote second peripheral regions configured of large-size pixels. Thoughnot illustrated, the pixel region divided into three or more phases ofperipheral regions can be configured by use of four or more kinds ofpixels in different sizes.

In a case where the pixel region is divided into a plurality of phasesof peripheral regions as illustrated in FIG. 20, the pixel region isbasically configured such that a farther region from the center regionhas a lower resolution (in other words, a higher resolution and a higherreading speed). For example, as the vehicle sped increases, the centerregion may be reduced, the peripheral regions may be set at a highersensitivity or a higher frame rate, or the number of divisions ofperipheral regions may be increased.

Further, the shape (contour) of each divided region is not limited tocircular. FIG. 21 illustrates an exemplary configuration of a pixelregion 210 which is divided into a center region 211, a first peripheralregion 212, and second peripheral regions 213 and 214, which arerectangular, respectively.

Additionally, the regions with a high sensitivity (or a low resolution)are drawn in thick gray in FIG. 20 and FIG. 21.

The imaging device in which the center region is leftward or rightwardfrom the center of the pixel region illustrated in FIG. 17 or FIG. 19may also be divided into two or more phases of peripheral regionssimilarly to the example illustrated in FIG. 20 or FIG. 21, and theshape of each region may be other than circular.

Also in the case of an imaging device divided into regions in adifferent way from those in FIG. 8 to FIG. 10, the shooting conditions(exposure time, sensitivity, frame rate, and reading speed) of eachphase of peripheral region may be controlled depending on a drivingsituation (such as fast traveling, going-through/traveling downtown,traveling during the nighttime or in a dark place, appearance of amoving object, abnormal driving, or traveling in tunnel) of the vehicleas indicated in Table 1.

In a dedicated imaging device in which pixels in a different size arearranged per region, the arrangement of a center region and peripheralregions is fixed. To the contrary, in a case where a method for forminga center region and peripheral regions by use of a signal processingsuch as gain control or pixel addition is employed in an imaging devicein which uniform-size pixels are arranged (see B-4), the signalprocessing is independently switched per region or in units of pixel,thereby flexibly and dynamically changing the position, shape, and sizeof each region. Further, the number of pixels to be subjected to pixeladdition is changed, thereby forming any number of phases of peripheralregions as illustrated in FIG. 20 or FIG. 21 by way of example. In otherwords, in consideration of design or productive efficiency of pixels oron-chip lenses, the imaging device in which uniform-size pixels arearranged is more excellent than the dedicated imaging device.

In the method for applying the signal processing to the imaging devicein which uniform-size pixels are arranged, the position of the centerregion can be moved leftward or right as illustrated in FIG. 17 or FIG.19 depending on a place or an orientation in which the shootingapparatus is installed, for example. Further, the position, size, andshape of the center region or peripheral regions, and the number ofphases of peripheral regions can be determined depending a drivingsituation (such as fast traveling, going-through/traveling downtown,traveling during the nighttime or in a dark place, appearance of amoving object, abnormal driving, or traveling in a tunnel) of thevehicle, and the shooting conditions (exposure time, sensitivity, framerate, and reading speed) of each region can be adaptively anddynamically changed as indicated in Table 1.

For example, when the vehicle approaches a left-hand curve 2301 asillustrated in FIG. 23A, an image shot in front of the vehicle (in thetraveling direction of the vehicle) by the shooting apparatus installedalmost at the center of the front node of the vehicle faster moves inthe landscape (object) on the right side of the pixel regioncorresponding to the outer periphery of the curve 2301, and blur easilyoccurs therein. As the vehicle speed increases at the curve 2301, bluris more remarkable. Thus, it is preferable to change region division ofthe pixel region such that a center region 2311 is leftward from thecenter of the pixel region as illustrated in FIG. 23B. Consequently, aleft peripheral region 2312 is narrower and a right peripheral region2313 is wider. Further, the appropriate shooting conditions are notnecessarily the same between the right and left peripheral regions. Theright peripheral region 2313 may be set at a shorter exposure time and ahigher frame rate than the left peripheral region 2312.

Whenever the steering angle of the steering wheel exceeds a certainangle, the center region 2311 may be stepwise shifted leftward. Further,as the vehicle speed increases, the amount of shift in the center regionmay be increased or decreased, the peripheral regions may be set at ahigher sensitivity or a higher frame rate, or the number of divisions ofperipheral regions may be increased.

As the vehicle speed increases, an object on the outer periphery of thecurve 2301 faster moves. Thus, it is preferable that a center region2321 is smaller, the peripheral regions are divided in multi-phases (twophases of peripheral regions 2323 and 2324 in the illustrated example),and the outer peripheral region 2324 is set at a higher sensitivity anda higher reading speed (or a lower resolution) thereby to keep theobject recognition rate of a moving object on the edge of the pixelregion as illustrated in FIG. 23C. When the vehicle speed exceeds acertain value at the curve 2301, the peripheral regions may be set at ahigher sensitivity and a higher frame rate, or the peripheral regionsmay be divided into multi-phases.

Further, when the vehicle approaches a right-hand curve 2401 asillustrated in FIG. 24A, an image shot in front of the vehicle (in thetraveling direction of the vehicle) by the shooting apparatus installedalmost at the center of the front nose of the vehicle faster moves inthe landscape (object) on the right side of the pixel regioncorresponding to the outer periphery of the curve 2401, and blur easilyoccurs therein. As the vehicle speed increases at the curve 2401, bluris more remarkable. Thus, it is preferable to change region division ofthe pixel region such that a center region 2411 is rightward from thecenter of the pixel region as illustrated in FIG. 24B. Consequently, aright peripheral region 2413 is narrower and a left peripheral region2412 is wider. Further, the appropriate shooting conditions are notnecessarily the same between the right and left peripheral regions. Theleft peripheral region 2412 may be set at a shorter exposure time and ahigher frame rate than the right peripheral region 2413.

Whenever the steering angle of the steering wheel exceeds a certainvalue, the center region 2411 may be stepwise shifted rightward.Further, as the vehicle speed increases, the amount of shift of thecenter region may be increased or decreased, the peripheral regions maybe set at a higher sensitivity or a higher frame rate, or the number ofdivisions of peripheral regions may be increased.

As the vehicle speed increases, an object on the other periphery of thecurve 2401 faster moves. Thus, it is preferable that a center region2421 is smaller, the peripheral regions are divided into multi-phases(two phases of peripheral regions 2422 and 2423 in the illustratedexample), and the outer peripheral region 2423 is set at a highersensitivity and a higher reading speed (or a lower resolution), therebykeeping the object recognition rate of a moving object at the edge ofthe pixel region as illustrated in FIG. 24C. When the vehicle speedexceeds a certain value at the curve 2401, the peripheral regions may beset at a higher sensitivity and a higher frame rate, or the peripheralregions may be divided in multi-phases.

For example, it is possible to measure an approach of the vehicle to aright-hand or left-hand curve or the vehicle speed at the time on thebasis of the steering angle of the steering wheel, the enginerevolutions, the rotation speed of the wheels, or the like detected bythe vehicle state detection part 2110, thereby performing adaptivecontrol of region division as illustrated in FIGS. 23A, 23B, 23C, 24A,24B and 24C.

Further, when the vehicle approaches an upward slope 2501 as illustratedin FIG. 25A, a vanishing point of the upward slope 2501 in the region ofgaze of the driver shifts upward from the center of the screen in animage shot in front of the vehicle (in the traveling direction of thevehicle) by the shooting apparatus installed almost at the center of thefront nose of the vehicle, and the road is largely shot. Alternatively,the driver's eyes tend to direct upward. Therefore, it is preferable toshift and widen the center position of a center region 2511 upward alongthe upward slope 2501 as illustrated in FIG. 25B.

On the other hand, when the vehicle approaches a downward slope 2601 asillustrated in FIG. 26A, a vanishing point of the downward slope 2601 inthe region of gaze of the driver shifts downward from the center of thescreen in an image shot in front of the vehicle (in the travelingdirection of the vehicle) by the shooting apparatus installed almost atthe center of the front nose of the vehicle, and the road rapidlydisappears. Alternatively, the driver's eyes tend to direct downward.Therefore, it is preferable to shift the center position of a centerregion 2611 downward along the downward slope 2601 as illustrated inFIG. 26B.

For example, it is possible to determine whether or not the vehicle istraveling on a slope on the basis of an angular speed (mainly pitchrate) of axial rotation of the vehicle body detected by the vehiclestate detection part 2110, an object recognition result on a road in animage shot by the vehicle-mounted camera, map information or roadinformation acquired in the navigation system included in the instrumentpanel 2730, and the like, thereby performing adaptive control of regiondivision as illustrated in FIGS. 25A, 25B, 26A, and 26B.

Exemplary adaptive control of region division illustrated in FIGS. 23A,23B, 23C, 24A, 24B, 24C, 25A, 25B, 26A, and 26B may define a regionincluding a vanishing point as a center region. Therefore, a vanishingpoint of the vehicle-mounted camera is tracked over time on the basis ofthe steering angle of the steering wheel, a measurement result of thetilt around the pitch axis of the vehicle body, or the like, therebyshifting the center region to include the vanishing point. Further, thesize or shape of the center region may be changed depending on thevehicle speed or other driving situation.

Further, there may be a method for defining a region including a pointof gaze of the driver (or eye direction) as a center region. Forexample, the center position of a center region is dynamically (overtime) shifted and its outer peripheral regions are also shifted tofollow the point of gaze of the driver sensed on the basis of an imageshot by the Dramoni camera included in the vehicle interior informationdetection part 2510. Alternatively, instead of shifting the centerregion according to the point of gaze of the driver, the vicinity of thepoint of gaze may be set at a higher resolution (or returned to asimilar resolution to the center region instead of performing pixeladdition) when the point of gaze of the driver moves toward a peripheralregion. This is because an object of interest is present in the eyedirection of the driver and may have to be recognized at a highresolution.

Further, when the headlamps are lit during the nighttime (includingcloudy weather or rainy weather) or in a dark place, the center regionirradiated by the headlamps can be clearly shot but the lights emittedfrom the headlamps do not reach the peripheral regions as previouslydescribed with reference to FIG. 7. Thus, the peripheral regions areshot at a high sensitivity due to pixel addition, and in a long exposuretime.

Further, region division of the pixel region of the imaging device maybe adaptively controlled depending on switching between high beam andlow beam in the body system control unit 2200 while the headlamps arelit.

When the headlamps are switched to high beam, a high-luminance regionirradiated by the headlamps shifts upward from the center of the pixelregion as illustrated in FIG. 27A. Thus, it is preferable that thecenter position of a center region 2701 is shifted upward and theportion where the lights emitted from the headlamps does not reach isassumed as a peripheral region 2702 and can be shot at a high resolutionas illustrated in FIG. 27B.

To the contrary, when the headlamps are switched to low beam, ahigh-luminance region irradiated by the headlamps shifts downward fromthe center of the pixel region as illustrated in FIG. 28A. Thus, it ispreferable that the center position of a center region 2801 is shifteddownward and the portion where the lights emitted from the headlamps donot reach is assumed as a peripheral region 2802 and can be shot at ahigh resolution as illustrated in FIG. 28B.

For example, the adaptive control of region division as illustrated inFIGS. 27A, 27B, 28A, and 28B can be performed in association withswitching control between high beam and low beam of the headlamps by thebody system control unit 2200.

FIG. 30 illustrates a flowchart of a processing procedure for performingregion division of a pixel region and shooting condition control of eachregion (peripheral region) depending on a driving situation in ashooting apparatus in which a center region and peripheral regions areprovided in a pixel region by use of a signal processing such as pixeladdition. The illustrated processing procedure assumes to be applied toa shooting apparatus in which uniform-size pixels are arranged, but canbe performed also in a shooting apparatus in which each region isconfigured of pixels in a different size. Further, the integratedcontrol unit 2600 in the vehicle control system 2000 mainly executespredetermined programs, for example, so that the illustrated processingprocedure is realized.

At first, a current driving situation (such as fast traveling,going-through/traveling downtown, traveling during the nighttime or in adark place, appearance of a moving object, abnormal driving, ortraveling in a tunnel) of the vehicle is grasped on the basis of adetection result of at least one of the vehicle state detection part2110, the vehicle exterior information detection part 2420, and thevehicle interior state detection part 2510, an analysis result of animage shot by the shooting part 2410, or the like (step S3001).

Then, the position, shape, and size of the center region suitable forthe driving situation are determined (step S3002).

Then, the number of phases of peripheral regions and the position,shape, and size of each phase of peripheral region, which are suitablefor the driving situation, are determined (step S3003).

Then, the shooting conditions (exposure condition, sensitivity, framerate, and reading speed) of each phase of peripheral region suitable forthe driving situation are determined on the basis of Table 1, forexample (steps S3004 and 3005).

The exposure processing is then performed on the vehicle-mounted cameraunder the determined shooting conditions (step S3006).

The recognition processing is further performed on an image shot by thevehicle mounted camera as described above (step S3007), and vehicledriving control may be performed on the basis of a recognition result ofthe peripheral regions, or the like (step S3008). Driving control willbe described below in detail.

B-8. Method for Using Image of Peripheral Region

According to the technology disclosed in the present specification, itis possible to improve the object recognition rate of peripheral regionsin an image shot by the vehicle-mounted camera. An object recognized ina peripheral region can be used for prediction or avoidance of dangersuch as collision, or for driving support or driving control of thevehicle by monitoring or tracking the object.

The peripheral regions are set at a high sensitivity to be adaptivelyshot at a high frame rate or in a short exposure time so that therecognition rate of a road sign (road guidance) installed along a roador a road sign or lane drawn on a road can be enhanced. A recognitionresult of a road sign or the like can be used for safe driving supportor cruise control such as lane deviation alarm, traveling speed control,and passing control.

Further, the peripheral regions are set at a high sensitivity to beadaptively shot at a high frame rate or in a short exposure time so thatthe recognition rate of a pedestrian, a crosswalk, or an obstacle cominginto a peripheral region can be enhanced. The recognition result may bedisplayed inside the vehicle by use of a head-up display, instrumentpanel, or the like, or may be output in speech, thereby warning of apedestrian or an obstacle.

Exemplary driving control using an image of a peripheral region will belisted.

-   -   (1) Recognition of Oncoming Vehicle or Forward-Traveling Vehicle        Hazard alarm, inter-vehicle adjustment, braking control, passing        control, lane change notification    -   (2) Recognition of Bicycle or Two-Wheels    -   Running-out alarm, braking control, avoidance driving from        danger/collision, notification to user (driver)    -   (3) Recognition of Pedestrian, Recognition of Crosswalk    -   Running-out alarm, braking control, avoidance driving from        danger/collision    -   (4) Recognition of Lane    -   Lane deviation alarm, cruise control    -   (5) Recognition of Sign    -   Speed control, alarm    -   (6) Recognition of Road Guidance    -   Speed control, navigation    -   (7) Previous (Near-Miss) Recognition Leading to Accident    -   Self-recording (dashboard camera), operation of safety apparatus        (such as air-bag)

B-9. Shot Image Display Method

An image shot by the shooting apparatus in which a pixel region isdivided into regions can be displayed inside the vehicle by use of ahead-up display or instrument panel, for example, or displayed as arecorded image on an apparatus outside the vehicle.

Further, in a case where the technology disclosed in the presentspecification is applied to a motorcycle, an image shot by thevehicle-mounted camera (or information acquired from images ofperipheral regions or object recognition result of peripheral regions,for example) may be augmented reality (AR) displayed on the shield of ahelmet which the driver wears, or the like, for example.

It may be difficult to recognize which portion is the center region or aperipheral region in an image during image display. Particularly, asdescribed in B-7, it is remarkably difficult to grasp each dividedregion in a case where region division is adaptively controlled. Thus,in a case where an image shot by the shooting apparatus in which a pixelregion is divided into regions is displayed, an image may be presentedper divided region. Alternatively, it is desirable that the bordersbetween regions can be visually confirmed by blending the regions, forexample, when one image is displayed.

Further, a change in an image per frame is smaller in the center regionas described with reference to FIG. 3. Thus, a plurality of frames inthe center region may be combined and displayed. For example, aplurality of shot images of the center region are HDR-combined whilechanging exposure, thereby generating an image with a wide dynamicrange.

INDUSTRIAL APPLICABILITY

The technology disclosed in the present specification has been describedabove in detail with reference to specific embodiments. However, it isclear that those skilled in the art can modify or replace theembodiments without departing from the spirit of the technologydisclosed in the present specification.

The technology disclosed in the present specification can be applied toa vehicle-mounted camera installed in any eye direction at anyplace in avehicle in order to shoot around the vehicle (outside the vehicle), suchas front nose, side mirrors, rear bumper, or back door of the vehicle.Further, the technology disclosed in the present specification can beapplied to digital mirror cameras.

In addition, the technology disclosed in the present specification canbe applied to various vehicles such as automobile (including gasolinepowered car and diesel powered car), electric-powered car, electrichybrid car, motorcycle, bicycle, and personal mobility. Further, thetechnology disclosed in the present specification can be applied to ashooting apparatus mounted on a mobile object (such as air plane) otherthan vehicles traveling on roads, or a monitoring camera.

In short, the technology disclosed in the present specification has beendescribed by way of example, and the contents described in the presentspecification should not be limitedly interpreted. CLAIMS should bereferred to in order to determine the spirit of the technology disclosedin the present specification.

Additionally, the technology disclosed in the present specification cantake the following configurations.

(1) A shooting control apparatus including:

-   -   a control part configured to control shooting conditions of a        center region in a shooting part having a plurality of pixels to        be any of a higher sensitivity, a higher frame rate, a shorter        exposure time, and a higher operation frequency than peripheral        regions in the shooting part.

(2) The shooting control apparatus according to (1),

-   -   in which the control part sets the peripheral regions at a        higher sensitivity than the center region.

(3) The shooting control apparatus according to (1),

-   -   in which the control part sets the peripheral regions at a        shorter exposure time than the center region.

(4) The shooting control apparatus according to (1),

-   -   in which the control part sets the peripheral regions at a        longer exposure time than the center region.

(5) The shooting control apparatus according to (1),

-   -   in which the control part sets the peripheral regions at a        higher frame rate than the center region.

(6) The shooting control apparatus according to (1),

-   -   in which the control part performs a signal processing in the        peripheral regions at a higher operation frequency than in the        center region.

(7) The shooting control apparatus according to (1),

-   -   in which the control part sets the peripheral regions at a        higher sensitivity and a higher frame rate than the center        region.

(8) The shooting control apparatus according to (1),

-   -   in which the control part sets the peripheral regions at a        higher sensitivity than the center region and at the same frame        rate as the center region.

(9) The shooting control apparatus according to (1),

-   -   in which the control part sets the peripheral regions at a        higher sensitivity than the center region and at the same or a        longer exposure time as or than the center region.

(10) The shooting control apparatus according to any of (2) to (9),

-   -   in which the control part performs pixel addition reading or        thinning reading on the peripheral regions.

(11) The shooting control apparatus according to any of (1) to (10),

-   -   in which the control part controls shooting conditions of the        peripheral regions relative to the center region depending on a        place where the shooting part is mounted on a vehicle or a        driving situation of the vehicle.

(12) The shooting control apparatus according to any of (1) to (11),

-   -   in which the control part controls at least one of the position,        the shape, and the size of the center region depending on a        place where the shooting part is mounted on a vehicle or a        driving situation of the vehicle.

(13) The shooting control apparatus according to any of (1) to (12),

-   -   in which the control part controls at least one of the number of        phases of the peripheral regions, the position, the shape, and        the size of each peripheral region depending on a driving        situation of a vehicle mounting the shooting part thereon.

(14) A shooting control method including:

-   -   a control step of controlling shooting conditions of a center        region in a shooting part having a plurality of pixels to be any        of a higher sensitivity, a higher frame rate, a shorter exposure        time, and a higher operation frequency than peripheral regions        in the shooting part.

(15) A shooting apparatus including:

-   -   an imaging device including a center region, and peripheral        regions configured of larger-size pixels than the center region.

(16) The shooting apparatus according to (15),

-   -   in which each of the center region and the peripheral regions is        scanned in parallel.

(17) The shooting apparatus according to (15), further including:

-   -   a signal processing part configured to perform pixel reading and        an AD conversion processing for each of the center region and        the peripheral regions.

(18) The shooting apparatus according to (15),

-   -   in which the peripheral regions are set at a shorter exposure        time than the center region.

(19) The shooting apparatus according to (15),

-   -   in which the peripheral regions are set at a higher frame rate        than the center region.

(20) The shooting apparatus according to (15),

-   -   in which at least one of an exposure time or a frame rate of the        peripheral regions is controlled relative to the center region        depending on a driving situation of a vehicle mounting the        shooting apparatus thereon.

REFERENCE SIGNS LIST

-   2000 Vehicle control system-   2010 Communication network-   2100 Drive system control unit-   2110 Vehicle state detection part-   2200 Body system control unit-   2300 Battery control unit-   2310 Battery apparatus-   2400 Vehicle exterior information detection unit-   2410 Shooting part-   2420 Vehicle exterior information detection part-   2500 Vehicle interior information detection unit-   2510 Vehicle interior state detection part-   2600 Integrated control unit-   2610 Microcomputer-   2620 General-purpose communication interface-   2630 Dedicated communication interface-   2640 Positioning part-   2650 Beacon reception part-   2660 In-vehicle device interface-   2670 Speech/image output part-   2680 Vehicle-mounted network interface-   2690 Storage part-   2710 Audio speaker-   2720 Display part-   2730 Instrument panel-   2760 In-vehicle device-   2800 Input part-   2900 Vehicle-   2910, 2912, 2914, 2916, 2918 Shooting part-   2920, 2922, 2924 Vehicle exterior information detection part-   2926, 2928, 2930 Vehicle exterior information detection part-   3100 Camera module-   3110 Camera control part-   3111 Shooting lens-   3120 Imaging device-   3130 Image processing part-   3140 Phase difference detection part-   3150 Display processing part-   3160 Display part-   3170 Image output part-   3180 Image recording control part-   3210 Timing control circuit-   3220 Row scanning circuit-   3230 Transfer signal generation circuit-   3240 Pixel array part-   3241 Phase difference pixel-   3242 Normal pixel-   3250 D/A conversion part-   3260 A/D conversion part-   3262 Comparator-   3263 Memory-   3270 Counter-   3290 Column scanning circuit

The invention claimed is:
 1. A shooting control apparatus, comprising: acontrol part configured to: determine a driving situation of a vehicleon which a shooting part is mounted, wherein the shooting part includesa center region and a plurality of peripheral regions; change a numberof the plurality of peripheral regions and at least one of an exposuretime or a frame rate of the plurality of peripheral regions relative tothe center region of the shooting part based on the determined drivingsituation, wherein the plurality of peripheral regions is divided into aplurality of regions each having a different pixel size; and controlshooting conditions of each region of the plurality of regions based onthe determined driving situation.
 2. The shooting control apparatusaccording to claim 1, wherein the control part is further configured toset the plurality of peripheral regions at a higher sensitivity than thecenter region.
 3. The shooting control apparatus according to claim 1,wherein the control part is further configured to set the plurality ofperipheral regions at a shorter exposure time than the center region. 4.The shooting control apparatus according to claim 1, wherein the controlpart is further configured to set the plurality of peripheral regions ata longer exposure time than the center region.
 5. The shooting controlapparatus according to claim 1, wherein the control part is furtherconfigured to set the plurality of peripheral regions at a higher framerate than the center region.
 6. The shooting control apparatus accordingto claim 1, wherein the control part is further configured to execute asignal processing in the plurality of peripheral regions at a higheroperation frequency than in the center region.
 7. The shooting controlapparatus according to claim 1, wherein the control part is furtherconfigured to set the plurality of peripheral regions at a highersensitivity and a higher frame rate than the center region.
 8. Theshooting control apparatus according to claim 1, wherein the controlpart is further configured to: set the plurality of peripheral regionsat a higher sensitivity than the center region; and set the plurality ofperipheral regions at a same frame rate as the center region.
 9. Theshooting control apparatus according to claim 1, wherein the controlpart is further configured to: set the plurality of peripheral regionsat a higher sensitivity than the center region; and set the plurality ofperipheral regions at one of a same exposure time as the center regionor a longer exposure time as or than the center region.
 10. The shootingcontrol apparatus according to claim 2, wherein the control part isfurther configured to execute pixel addition reading or thinning readingin the plurality of peripheral regions.
 11. The shooting controlapparatus according to claim 1, wherein the control part is furtherconfigured to control shooting conditions of the plurality of peripheralregions relative to the center region based on one of a place where theshooting part is mounted on the vehicle or the determined drivingsituation of the vehicle.
 12. The shooting control apparatus accordingto claim 1, wherein the control part is further configured to control atleast one of a position, a shape, or a size of the center region basedon one of a place where the shooting part is mounted on the vehicle orthe determined driving situation of the vehicle.
 13. The shootingcontrol apparatus according to claim 1, wherein the control part isfurther configured to control at least one of a number of phases of theplurality of peripheral regions, a position, a shape, or a size of eachperipheral region of the plurality of peripheral regions based on thedetermined driving situation of the vehicle.
 14. A shooting controlmethod, comprising: determining a driving situation of a vehicle onwhich a shooting part is mounted, wherein the shooting part includes acenter region and a plurality of peripheral regions; changing a numberof the plurality of peripheral regions and at least one of an exposuretime or a frame rate of the plurality of peripheral regions relative tothe center region of the shooting part based on the determined drivingsituation, wherein the plurality of peripheral regions is divided into aplurality of regions each having a different pixel size; and controllingshooting conditions of each region of the plurality of regions based onthe determined driving situation.
 15. A shooting apparatus, comprising:an imaging device includes: a center region; and a plurality ofperipheral regions, wherein the plurality of peripheral regions haslarger-size pixels than the center region, a number of the plurality ofperipheral regions is changed based on a driving situation of a vehicleon which the shooting apparatus is mounted, at least one of an at leastone of an exposure time or frame rate of the plurality of peripheralregions is changed relative to the center region of the imaging device,the plurality of peripheral regions is divided into a plurality ofregions each having a different pixel size, and shooting conditions ofeach region of the plurality of regions is controlled based on thedriving situation.
 16. The shooting apparatus according to claim 15,wherein each of the center region and the plurality of peripheralregions is scanned in parallel.
 17. The shooting apparatus according toclaim 15, further comprising a signal processing part configured toexecute a pixel reading process and an AD conversion process for each ofthe center region and the plurality of peripheral regions.
 18. Theshooting apparatus according to claim 15, wherein the plurality ofperipheral regions is set at a shorter exposure time than the centerregion.
 19. The shooting apparatus according to claim 15, wherein theplurality of peripheral regions is set at a higher frame rate than thecenter region.